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ABSTRACT 



The career academy approach is one of the oldest and most 
widely established high school reforms in the U.S., stretching back for more 
than 30 years. A large-scale, multi-site, random assignment research design 
was conducted to determine the impact of career academies on student 
outcomes. Some of the findings of the study include the following: (1) the 

career academies in the study increased both the level of interpersonal 
support students experienced and their participation in career awareness and 
work-based learning activities; (2) the career academies substantially 
improved high school outcomes among students at high risk of dropping out; 

(3) among students least likely to drop out, the career academies increased 
the likelihood of graduating on time; (4) in academies that greatly enhanced 
interpersonal support from teachers and peers, the dropout rates dropped 
dramatically; (5) the academies did not improve standardized mathematics and 
reading test scores; and (6) the impact of the academies and the types of 
students who participated in them varied greatly from site to site. The study 
concluded that career academies can be an effective means of reducing the 
high school dropout rate and enhancing students' engagement with school, 
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especially if they increase personal support of students through involvement 
with teachers and peers. (Contains 33 references and two appendixes that 
provide supplementary information about the career academies research study 
and strategies for creating subgroups of students defined by at-risk 
characteristics.) (KC) 
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Preface 



Career Academies have existed for more than 30 years, and they can now be found in an 
estimated 1,500 high schools nationwide. The durability and broad appeal of the Career Acad- 
emy approach can be attributed, in part, to the fact that its core features offer direct responses to a 
number of problems that have been identified in large comprehensive high schools. Career 
Academies consist of small learning communities that aim to create a more personalized and 
supportive learning environment for students and teachers. They combine academic and career- 
related courses in an effort to enhance both the rigor and the relevance of the high school curricu- 
lum. Career Academies form partnerships with local employers to expand students’ exposure to 
career options and skills requirements and to provide them with work-based learning experiences. 
The primary goals of the Career Academy approach are to enhance students’ engagement and 
performance in high school and provide them with the credentials and skills needed to make suc- 
cessful transitions to post-secondary education and, eventually, a career. 

This report from MDRC’s ongoing Career Academies Evaluation is being released at a 
time when education policymakers and practitioners are pursuing a number of far-reaching 
strategies for improving American high schools. Many of these strategies include principles em- 
bedded in the Career Academy approach, while others include the Career Academy model as an 
explicit component. In short, Career Academies stand at the intersection of several major educa- 
tion reforms aimed at transforming high schools into nurturing, productive places where students 
leam and grow and are prepared for careers in an economy that demands high skills and adapt- 
ability. 



In partnership with the funding organizations listed at the front of this report and with 
staff from the participating sites, MDRC began an in-depth evaluation of the Career Academy 
approach in 1993. The primary purpose is to provide policymakers and educators with reliable 
evidence about the extent to which the Academies deliver on their ambitious goals, outlined 
above. In particular, the evaluation provides a rigorous assessment of Career Academies’ effects 
on a range of education, developmental, and work-related outcomes for high school students. The 
study also offers lessons about how Career Academies operate and are sustained and about the 
pathways through which they affect students’ engagement and performance during high school 
and beyond. 

This report marks an important milestone in the Career Academies Evaluation. Previous 
reports and papers from the study described some distinctive features of the Career Academies, 
relative to their regular school environments, and examined some differences in the school- and 
work-based experiences of Academy and non-Academy students in the study sample. These re- 
ports concluded that the distinctive features of the participating Career Academies had indeed 
enhanced students’ experiences in school and the workplace. The current report provides evi- 
dence about the extent to which these enhancements translated into higher levels of school en- 
gagement and performance and whether Academy students are better prepared than their non- 
Academy peers to make the transition from high school to further education and the labor mar- 
ket. 



The findings suggest that a growing number of high schools may be on the right path to- 
ward keeping students engaged in school and preparing them for further education and a career. 
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Career Academies reduced dropout rates and improved school engagement among students least 
likely to do well in a regular school environment. While the Academies produced more modest 
effects for other students, they created a more supportive school environment for all students and 
provided them with more opportunities to explore careers and engage in work-based learning op- 
portunities. It is not yet clear how the Academies affect students as they navigate the transition 
between high school and college and the labor market. The evaluation will continue to follow 
students in the study sample to assess the Academies’ longer-term post-high school effects. 

The report draws on a particularly rigorous research design and an unusually rich database. 
This evaluation has demonstrated the feasibility and benefits of using a random assignment re- 
search design to determine the impact of Career Academies on student outcomes. A rarity in 
education research, this design provides an especially reliable way of comparing the performance 
of students who had access to an Academy with that of a truly comparable group of students who 
did not have access to the programs. The database consists of survey information provided by 
Academy and non-Academy students in the study sample, performance indicators obtained from 
school records and transcripts, and standardized test scores from a test MDRC administered to a 
sample of the students. The report also draws on qualitative information collected during the many 
field visits to each of the participating sites over the past seven years and through ongoing commu- 
nication with the staff in the sites. Students in the study sample were identified in the 8 th or 9 th 
grade, and this report follows them through the end of their scheduled 12 th -grade year until just 
before they would have graduated from high school. 

It is our hope that the Career Academies Evaluation will continue to offer policymakers 
and educators useful lessons about what works for high school students and about the value of 
subjecting promising school reform approaches to rigorous tests of their effectiveness. 



Judith M. Gueron 
President 
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Executive Summary 



The Career Academy approach is one of the oldest and most widely established high 
school reforms in the United States. Career Academies have existed for more than 30 years and 
have been implemented in more than 1 ,500 high schools across the country. The durability and 
broad appeal of the Academy approach can be attributed, in part, to the fact that its core features 
offer direct responses to a number of problems that have been identified in large comprehensive 
high schools. Career Academies attempt to create more supportive and personalized learning en- 
vironments through a school-within-a-school structure. Their curricula combine academic and 
occupation-related course requirements that aim both to promote applied learning and to satisfy 
college entrance requirements. Academies establish partnerships with local employers to build 
sequences of career awareness and work-based learning opportunities for their students. 

While the basic organizational features of the approach have remained the same since Ca- 
reer Academies’ inception, the goals and target population have changed. The original Acade- 
mies were designed primarily to prevent dropping out of high school and to increase preparation 
for work among students who began high school at high risk of school failure. There is now 
widespread agreement that Career Academies should seek to prepare students for both work and 
college, and that they should include a broad cross-section of students, including those who are 
highly engaged in school. 

There has been a great deal of research on the Academy approach. Nevertheless, previous 
studies have been unable to determine reliably whether differences between Academy students’ 
high school experiences and outcomes and those of other students result from the Academy itself 
or from the program’s student targeting or its selection practices. Further, little is known about 
the relative effectiveness of Academies for different groups within the broad cross-section of stu- 
dents they now serve. There have also been few opportunities to explore the extent to which dif- 
ferent contexts and implementation strategies may influence the effectiveness of the Academy 
approach. 

This evaluation has demonstrated the feasibility and benefits of using a large-scale, multi- 
site random assignment research design to determine the impact of Career Academies on student 
outcomes. A rarity in education research, this design provides a uniquely rigorous way of com- 
paring the performance of students who had access to an Academy with the performance of a 
truly comparable group of students who did not have access to the programs. In order to address 
a number of key policy issues for Academies and related high school reforms, this report focuses 
on three questions: 

• To what extent does the Career Academy approach alter the high school envi- 
ronment in ways that better support students academically and developmen- 
tally? 

• To what extent does the Career Academy approach change educational, em- 
ployment, and youth development outcomes for students at greater or lesser 
risk of school failure? 
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• How do the manner and context in which Career Academy programs are im- 
plemented influence their effects on student outcomes? 

This report marks a milestone in the Manpower Demonstration Research Corporation’s 
(MDRC) 10-year Career Academies Evaluation, which is being supported by the U.S/Depart- 
ments of Education and Labor and by 17 private foundations and organizations. The report fo- 
cuses on over 1,700 students who had applied for one of nine Career Academies participating in 
the evaluation. 1 The participating Academies were able to implement and sustain the basic fea- 
tures of the approach and have adapted to a wide range of local needs and circumstances. They 
include a range of technical, service-oriented, and business-related career themes and are located 
in small cities and large urban school districts. Students in the study sample were identified when 
they were in 8 th or 9 th grade, and this report follows them through the end of their scheduled 12 th - 
grade year. 



Findings in Brief and Policy Implications 

Throughout this report, the term impact refers to differences between outcomes for stu- 
dents who were randomly selected to enroll in an Academy and those of students who also ap- 
plied but were not selected to enroll. Academy applicants were assigned to these groups at ran- 
dom, so there were no systematic differences in the characteristics or school experiences of the 
applicants initially. Thus, subsequent differences in outcomes for the two groups reflect increases 
or decreases caused by the Career Academies. Following is a summary of the key findings from 
the report. 

• The Career Academies in this study increased both the level of interpersonal 
support students experienced during high school and their participation in ca- 
reer awareness and work-based learning activities. 

• The Career Academies substantially improved high school outcomes among 
students at high risk of dropping out. For this group, the Academies reduced 
dropout rates, improved attendance, increased academic course-taking, and in- 
creased the likelihood of earning enough credits to graduate oh time. 

• Among students least likely to drop out of high school, the Career Academies 
increased the likelihood of graduating on time. The Academies also increased 
vocational course-taking for these students without reducing their likelihood 
of completing a basic core academic curriculum. 

• In sites where the Academies produced particularly dramatic enhancements in 
the interpersonal support that students received from teachers and peers, the 
Career Academies reduced dropout rates and improved school engagement for 
both high-risk and medium-risk subgroups (about 75 percent of the students 



'Ten sites were initially selected for the evaluation. One of the initial Career Academies was disbanded after 
two years in the study and was unable to provide sufficient follow-up data to be included in the impact analysis for 
this report. 



served). Academies that- did not enhance these supports actually increased 
dropout rates and reduced school engagement for some students. 

• The Career Academies did not improve standardized math and reading 
achievement test scores. 

• When the findings are averaged across the diverse groups of students in the 
full study sample, it appears that the Career Academies produced only slight 
reductions in dropout rates and modest increases in other measures of school 
engagement. These aggregated findings, however, mask the high degree of 
variation in effectiveness among different groups of students and across the 
different program contexts. 

The findings that have emerged from the Career Academies Evaluation to date suggest 
the following implications for policies aimed at improving high schools and helping students 
prepare for the transition from high school to further education and work. 

• Career Academies provide a well-defined approach to creating more suppor- 
tive high school environments and increasing students’ exposure to career 
awareness and work-based learning activities. ' 

• Among students who are most at risk of dropping out of high- school, Career 
Academies are an effective means of preventing dropout, increasing school 
engagement, and helping students acquire the credentials they need to gradu- 
ate and prepare for post-secondary education. 

• Career Academies should continue to serve a heterogeneous population of 
students. The pervasive positive impacts for students at high risk of dropping 
out may derive, in part, from exposure to a highly engaged peer group who, on 
balance, also benefit from exposure to several key dimensions of the Academy 
experience. 

I 1 

• If Career Academies do not complement their career-related curriculum and 
work-based learning activities with strong interpersonal and academic sup- 
ports, they risk reducing school engagement for some students. A highly 
structured school-within-a-school organization can create a necessary set of 
conditions for providing these supports. 

• Career Academies should build on the effective organizational enhancements 
they bring to high school reform efforts if they are to improve academic 
achievement as measured by most standardized tests currently in use. Prom- 
ising approaches may involve aligning Career Academy curricula with high 
standards and providing teachers with the incentives and capacity to deliver on 
such standards. 

The above results capture the effects that the Career Academies have had on students 
through the end of their scheduled 12 th -grade year. The evaluation does not yet include informa- 
tion about the rates at which these students actually graduated from high school and whether the 
dropouts eventually returned to high school or pursued an alternative credential. The next phase 
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of this evaluation will include this information and will follow the students in the study sample 
for four additional years as they make the transition from high school to post-secondary educa- 
tion and employment opportunities. 

The remainder of this Executive Summary describes the Career Academy approach in 
greater detail, including its history, and discusses the current policy context and previous re- 
search in the Career Academies Evaluation. It then describes the results of the evaluation and 
their implications for policy and practice. 



The Career Academy Approach 

The Career Academy approach is distinguished by three core features that offer direct re- 
sponses to several problems that have been identified in high schools, particularly those serving 
low-income communities and students at risk of school failure. First, a Career Academy is or- 
ganized as a school-within-a-school in which students stay with a group of teachers over three or 
four years in high school. Such arrangements are often referred to as “small learning communi- 
ties.” The aim is to create a more personalized and supportive learning environment for students 
and teachers. Second, a Career Academy offers students a combination of academic and voca- 
tional curricula and uses a career theme to integrate the two. Third, a Career Academy estab- 
lishes partnerships with local employers in an effort to build connections between school and 
work and to provide students with a range of career development and work-based learning op- 
portunities. This definition of an Academy is now commonly accepted and was reviewed by a 
broad range of researchers, policymakers, and practitioners who have worked closely with Career 
Academies. 

The initial Career Academies of the 1970s and 1980s were primarily vocational education 
programs targeted at students who appeared to be at high risk of dropping out of high school. The 
central goals of these early programs were to keep students engaged in school, provide them with 
work-related learning experiences both in the classroom and on the job, and establish clearer 
pathways between high school and post-secondary employment. 

Since the late 1980s, there has been a shift in the primary goals and target population of 
most Career Academies. In particular, there is now wide agreement that the Career Academy ap- 
proach should be explicitly distinct from traditional vocational education by seeking to prepare 
students for both work and college. Vocational education, as defined in federal law and through 
its historical legacy, has been directed at preparing young people for occupations that do not tra- 
ditionally require advanced degrees. In line with what has been called “the new vocational edu- 
cation,” Career Academies now seek to include a broad range of students and to combine a rigor- 
ous academic curriculum with exposure to extensive information about an industry both in the 
workplace and in the classroom. The career theme is used to integrate curricula and provide ex- 
posure to a broad array of careers in a given field and does not typically focus on preparing stu- 
dents for jobs in those areas. 

The 1990s have seen extraordinary growth in the number of Career Academies around the 
country. There are estimated to be more than 1,500 Career Academies nationwide, representing 
nearly a 15-fold increase in approximately 10 years; many more Academies are in the planning 
stages. Much of this growth can be traced to the increasing number of national, state, and district 
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Academy support networks. Although most Career Academies share the approach’s basic ele- 
ments, the Academy model has been adapted to a wide range of local needs and circumstances, 
resulting in a variety of versions that emphasize different features over others. 

The expansion of Career Academy target populations and goals and the rapid growth in the 
number of Academies have raised several questions about how the Academy approach may be af- 
fecting students’ performance in high school. How well does it meet the needs of a much broader 
range of students than it was initially designed to serve? Is the Academy approach more effective 
under some conditions than under others? Which features of the Academy model make the most 
difference for students? MDRC’s Career Academies Evaluation is intended to shed light on these 
and other questions. 

The Current Policy Context 

This report is being released at a time when education policymakers and practitioners are 
pursuing a number of far-reaching strategies for improving American high schools. Many of 
these strategies include principles embedded in the Career Academy approach, while others in- 
clude the Career Academy model as an explicit component. Two key policy initiatives are par- 
ticularly relevant. 

First, states, school districts, and employers are now looking for strategies and ap- 
proaches that can build on the foundation established by the School-to-Work Opportunities Act 
(STWOA) of 1994. STWOA was aimed at enhancing the relevance and rigor of school- and 
work-based learning and at creating clearer pathways between high school and post-secondary 
education and careers. This was to be done primarily through partnerships between schools and 
local employers. STWOA specifically identifies Career Academies as a “preferred approach” to 
creating such partnerships and implementing the principles embedded in the legislation. STWOA 
is scheduled to sunset in 2001 — adding urgency to these efforts and heightening interest in con- 
crete evidence of the potential payoff of Career Academies. 

Second, the U.S. Department of Education has committed itself to several initiatives 
aimed specifically at addressing problems that are unique to high schools. Many of these initia- 
tives are being supported under the Comprehensive School Reform Demonstration developed 
within the Office of Educational Research and Improvement (OERI) and the New American 
High Schools established by the Office of Vocational and Adult Education (OVAE). Although 
most of the strategies that are being developed involve comprehensive reforms of entire high 
schools, many include key elements of the Academy approach, including the creation of a small 
school-within-a-school, integration of academic and vocational curricula, and the establishment 
of partnerships with employers and other organizations in the community. 

The findings presented in this report will shed light on the extent to which the Career 
Academy model, and some of the high school reform approaches embedded in the model, can 
achieve the goals sought by their proponents. 
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The Career Academies EvataMMIl 



In 1993, MDRC began an evaluation of the Career Academy approach as it had been de- 
fined in previous research and implemented in a broad range of settings across the country. The 
evaluation’s primary goal is to provide policymakers and educators with reliable evidence about 
the impact that Career Academies have on students’ success in high school and their transition to 
further education and the labor market. The evaluation will also offer lessons about how Career 
Academies operate and are sustained and about the pathways through which Academies affect 
student engagement and performance in school. 

The current report is based on information collected over a six-year period and focuses on 
nine high schools and their Career Academies. 2 Each of the Academies had established the basic 
Career Academy components described earlier: a school-within-a-school organization, an 
integrated academic/vocational curriculum, and employer partnerships. Moreover, this combination 
of features was not available elsewhere in the participating high schools. 3 These Academies were 
selected to include school districts and high schools reflecting the typical conditions (large urban 
centers and small cities) under which Career Academies have been implemented across the country. 
MDRC was specifically interested in Academies serving a broad range of students, including those 
who were perceived to be at risk of not succeeding in the regular high school environment. Most of 
the school districts in the evaluation are large and enroll substantially higher percentages of 
African-American and Hispanic students than school districts nationally. On average, these school 
districts have higher dropout rates, higher unemployment rates, and higher percentages of low- 
income families. 

The Career Academies Evaluation is a rarity in the field of education research in that it 
has demonstrated the feasibility and benefits of implementing a large-scale, multi-site random 
assignment research design within an ongoing high school program. This was made possible be- 
cause each of the Career Academies in the study received applications from approximately twice 
as many students as it was able to serve. This reports focuses on a sample of 1,764 students (re- 
ferred to in this report as the study sample) who applied for one of the Career Academies selected 
for the study. Of these, 959 students were randomly assigned to the program group (referred to in 
this report as the Academy group) and were accepted for admission to the Academies. The re- 
maining 805 students were randomly assigned to a control group (referred to in this report as the 
non-Academy group) and were not invited to participate in the Academies, although they could 
choose other options in the high school or school district. 

The random assignment process ensured that there were no systematic differences between 
the two groups of students in terms of their observable and unobservable background characteris- 
tics, prior school experiences, and initial motivation and attitudes toward school. Any systematic 



2 For a more detailed description of the criteria and process used to select sites for this study, see James J. Kem- 
ple and JoAnn Leah Rock, Career Academies: Early Implementation Lessons from a 10-Site Evaluation (New 
York: MDRC, 1996). 

Although some participating high schools do operate other programs that they classify as Career Academies, 
information collected for this study indicated that most such programs do not include all the basic components of 
the Academy approach described earlier. As a result, the participating Career Academy programs represent a clear 
contrast with the other programs in the high schools. 
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differences that subsequently emerged between the groups can be attributed with confidence to dif- 
ferences in their access and exposure to the Career Academies. 

MDRC obtained data for this report from four sources: 

o school transcript records, including information about students’ daily atten- 
dance rates, credits earned toward graduation, and course-taking patterns; 

o student surveys that asked a wide range of questions about school experi- 
ences, employment and work-related experiences, extracurricular activities, 
preparation for college and post-secondary jobs, and plans for the future; 

o standardized math computation and reading comprehension tests admin- 
istered to 490 students from the study’s sample (from both the Academy and 
the non-Academy groups) at the end of their n^-grade year; 4 and 

o qualitative field research conducted throughout the evaluation to document 
Academies’ characteristics, local contexts, staff, students, and employer part- 
ners. 

Students in the study sample were identified at the end of 8 th or 9 th grade. This report fol- 
lows them for three or four years through the end of their scheduled 12 th -grade year, until just 
before they would have graduated from high school. The primary focus of the report is on out- 
comes measured at the end of students’ scheduled ^-grade year. Unless otherwise noted, the 
impact findings discussed in the report are statistically significant, indicating that one may have a 
high degree of confidence that measured differences in outcomes between the Academy and the 
non-Academy groups were not a result of chance. 



Previously Reported FimidlDinigs oiro How Caireeir Academies Ghaumged Stiudeiniis’ 
High School Experiences 

The previous reports from this evaluation examined the extent to which Career Acade- 
mies changed the high school environment as indicated by differences between Academy and 
non-Academy students’ experiences during high school. 5 Following is a brief overview of key 
findings from these reports. 

o The Career Academies enhanced the degree of interpersonal support stu- 
dents received from teachers and peers. 



4 The test instrument comprised the reading comprehension and math test batteries from the National Educa- 
tional Longitudinal Survey of 1988 (NELS: 88) Follow-up Study. A total of 490 students from the study sample 
completed the test, including both high school dropouts and students who remained enrolled in school. 

5 For a more detailed discussion of these findings, see James J. Kemple, Career Academies: Communities of 
Support for Students and Teachers: Emerging Findings from a 10-Site Evaluation (New York: MDRC, 1997); 
James J. Kemple, “Selected Dimensions of Applied Learning in Career Academy Classrooms,” unpublished MDRC 
paper, 1997; and James J. Kemple, Susan M. Poglinco, and Jason C. Snipes, Career Academies: Building Career 
Awareness and Work-Based Learning Activities Through Employer Partnerships (New York: MDRC, 1999). 
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During their early years in high school, Academy students received more support from 
their teachers and peers than did their counterparts in non-Academy high school environments. 
For example, compared with their non-Academy peers, Academy students reported that their 
teachers had higher expectations of them and that teachers provided them with more individual- 
ized attention. Moreover, compared with their non-Academy counterparts, Academy students 
were more likely to report that their classmates were highly engaged in school and that they had 
many opportunities to collaborate with their peers on school and work-related projects. 

• Career Academies increased students’ participation in career awareness 
and work-based learning experiences during high school. 

Academies also increased students’ exposure to work-related learning experiences in 
school and in the workplace. Academy students were more likely than their non-Academy peers 
to be exposed to career-related themes or activities in the classroom. They were also more likely 
to participate in career-related activities such as job-shadowing or field trips. Finally, Academy 
students were more likely than non-Academy students to participate in a planned program of 
work experience and to have high-quality work-based learning experiences during high school. 

® The Career Academies in this evaluation demonstrated their capacity to 
attract large numbers of applicants and to include students with a wide 
range of demographic and education characteristics. 

The growth of the Career Academy movement has been accompanied by questions about 
whether the programs can and should serve a broad range of students and about which students 
benefit most from them. Reflecting the shift in goals and target populations of Career Academies 
nationwide, the programs in this evaluation attracted a mix of students including those at risk of 
dropping out of high school or failing academically as well as students who had done well in 
school. Most of the students in the study sample are from minority backgrounds — 56 percent 
are Hispanic, and 30 percent are African-American — reflecting the racial and ethnic make-up of 
their communities. Also, more than one-third of the students came from single-parent house- 
holds, and about one-quarter indicated that their families received public assistance. At the same 
time, just under half the students reported that both their parents were employed, and about one- 
third reported that at least one parent had attended college. 

• Approximately 88 percent of the students selected for admission to a Ca- 
reer Academy actually enrolled in the programs, and 58 percent of those 
selected remained in an Academy throughout high school. 

Of the students who were initially selected for admission, about 12 percent chose not to 
enroll, and another 30 percent enrolled in the programs and then left before the end of their 12 th - 
grade year. It is unclear how much of this attrition could possibly be controlled or avoided by the 
Career Academies. Student mobility and early dropout are common in most urban school dis- 
tricts, and they were reasons for attrition from the Academies in this evaluation. Just under one- 
quarter of the students who never enrolled in an Academy or who enrolled and then left reported 
that they did so because their families moved and they had to transfer to other schools. Another 
20 percent reported that they were asked to leave the programs or dropped out of high school al- 
together. The remaining students (approximately 55 percent of those who were not enrolled in an 
Academy in the 12 th grade) chose not to enroll or chose to leave the programs. The most common 
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reasons students gave for not enrolling or for not remaining enrolled in an Academy were that 
they wanted to enroll in another program, they lost interest in the occupational area, or they did 
not think the Academy would help them get into a good college. 



The Impact of Career Academies orn Stmdemt Outcomes 

The central theme that has emerged from the Career Academies Evaluation thus far is that 
the Academies affected the outcomes for students who were likely to drop out of high school 
much more than they affected the outcomes for other students. When the results are averaged 
across the diverse groups of students they serve, it appears that the Academies produced only 
slight reductions in dropout rates, modest improvements in students’ progress toward high school 
graduation, and increases in career-related course-taking and involvement in positive youth de- 
velopment activities. These aggregate results mask a high degree of variation in the Career 
Academies’ potential to make a difference and in the actual differences they made for some stu- 
dents. 



To assess this variation in impacts, the study sample was divided into three subgroups 
based on selected background characteristics and prior school experiences. These characteristics 
were chosen as indicators of students’ engagement in school at the time they applied for an 
Academy and as factors associated with the likelihood of their eventually dropping out of school. 
(See Table ES-1 for a list of the background characteristics used to define these subgroups.) Just 
over one-quarter of the students were classified as being in the high-risk subgroup and reflected 
the combination of characteristics associated with the highest probability of dropping out among 
those in the non-Academy group. Approximately one-quarter of the students in the sample were 
classified as being in the low-risk subgroup and reflected the combination of characteristics asso- 
ciated with the lowest probability of dropping out among those in the non-Academy group. The 
remaining students (approximately half the sample) were defined as being in the medium-risk 
subgroup. 

Because each of the characteristics used to define the subgroups was measured before 
students were randomly assigned to the two main study groups, there are no systematic differ- 
ences in observed background characteristics between Academy and non-Academy groups 
within each of the three risk subgroups. 6 The following sections summarize the impact findings 
for these subgroups. 



6 The definition of these subgroups involved analyses using background characteristics to predict dropping out 
among students in the non-Academy group. This generated an index of average characteristics of likely dropouts 
who did not have access to an Academy. The index was then calculated for the Academy group using the same 
characteristics. Because the predicted relationship between background characteristics and dropout rates was based 
on the non-Academy group, however, it is likely to yield somewhat more accurate predictions of likely dropouts for 
that group than for the Academy group. This means that the dropout rate for the students in the high-risk non- 
Academy group may be artificially high. Extensive analyses were conducted to identify the potential magnitude of 
this distortion. These analyses indicate that whatever distortion exists is minimal and could not have changed the 
pattern of impacts. This issue is discussed in greater detail in Appendix B of the report. 
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Table ES-1 

Career Academies Evaluation 



Background Characteristics of Students, 
by Subgroups Defined by Risk of Dropping Out of School 



Characteristic 


High-Risk 

Subgroup 

(%) 


Medium-Risk 

Subgroup 

(%) 


Low-Risk 

Subgroup 

(%) 


Characteristics associated with dropping out of school 

Attendance rate, year prior to random assignment 


96-100% . 


24.4 


52.5 


91.0 


91-95% 


23.6 


32.3 


7.9 


86-90% 


18.7 


1 1.7 


1.0 


85% or lower 


33.3 


3.6 


0.2 


Credits earned in 9 tn grade 3 


5 or more credits . 


47.2 


93.8 


100.0 


3-4 credits 


35.1 


6.3 


0.0 


2 or fewer credits 


17.7 


0.0 


0.0 


Grade point average in year of random assignment 0 


3. 1 or higher 


12.5 


37.2 


58.6 


2. 1-3.0 


25.5 


44.0 


39.3 


2.0 or lower 


62.0 


18.7 


2.1 


Student is overage for grade level c 


43.0 


18.2 


2.4 


School mobility 0 

0 or 1 different school 


50.0 


71.9 


99.0 


2 or more different schools 


50.0 


28.1 


1.0 


Student has sibling who dropped out of high school 


42.7 


17.5 


0.5 


Sample size (N= 1,764) 


474 


869 


421 



SOURCES: MDRC calculations from the Career Academies Evaluation Student Baseline Questionnaire and Student 
School Records Databases. 



NOTES: All characteristics were measured at the time students applied to the Career Academy program and prior to 
being randomly selected to the Academy and non-Academy groups. 

Invalid or missing^values are not included in individual variable distribution. Rounding may cause slight 
discrepancies in calculating of sums and differences. 

Statistical significance tests are not included. 

^his was applicable only to students who applied to the Career Academy at the end of their 9 th -grade year. 
b Grade point averages were converted to a standard 4.0 scale from 100-point or 5-point scales for some sites. 

C A student is defined as overage for grade at the time of random assignment if she or he turned 1 5 before the 

start of the 9 th grade, or 16 before the start of the 10 th grade. This indicates that the student was likely to have been 
held back in a previous grade. 

d School mobility is defined as the number of schools attended since the 1 st grade beyond the number expected to 
result from promotions in grade level or graduations. 
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Career Academy Impacts for Students in the High-Risk Subgroup 

As shown in Table ES-1, students in the high-risk subgroup entered the study with back- 
ground characteristics and prior school experiences indicating that they were disengaged from 
school. More than half had failed courses during the 9 th grade, and about one-third could be clas- 
sified as chronic absentees (having attendance rates lower than 85 percent). Most of these stu- 
dents had low grade point averages (2.0 or lower), and over 40 percent had been held back in a 
previous grade (as indicated by being overage for their current grade). 

Figure ES-1 provides a summary of the impact findings for students in the high-risk sub- 
group. It shows first that, without access to an Academy, a high percentage of non- Academy stu- 
dents in the high-risk subgroup had become even more disengaged from school. In all, 32 percent 
of these students dropped out of high school, and only 26 percent had earned sufficient credits to 
meet the district’s graduation requirements by the end of their scheduled 12 th -grade year. 

© Among students at high risk of school failure, Career Academies signifi- 
cantly cut dropout rates and increased attendance rates, credits earned 
toward graduation, and preparation for post-secondary education. 

Figure ES-1 shows that the Career Academies produced substantial improvements in 
many educational outcomes for students in the high-risk subgroup. In particular, while 32 percent 
of the non- Academy students in the high-risk subgroup dropped out of high school, 21 percent of 
the Academy students did so. This 1 1 percentage point difference represents a one-third reduc- 
tion in the dropout rate for the non-Academy group. This can be classified as a particularly large 
reduction in dropout rates. Reductions of this magnitude are rare for school-based interventions. 

The Academies also significantly increased average attendance throughout high school 
for students in the high-risk subgroup (not shown in Figure ES-1). Average attendance rates 
throughout high school were approximately 76 percent for students in the non-Academy group, 
compared with 82 percent for students in the Academy group. This amounts to an additional 1 1 
days of school per year over four years. 

Moreover, while 26 percent of the high-risk non-Academy group had earned enough 
credits to meet district graduation requirements, 40 percent of the students in the Academy group 
did so (an increase of over 50 percent beyond the non-Academy group average). This suggests 
that, besides improving attendance and preventing students from dropping out, the Academies 
helped a significant portion of the high-risk subgroup to make up enough of the initial gap in 
credits earned to meet the district’s graduation requirements three year later. 

Also, as indicated by the third set of bars in Figure ES-1, the Academies doubled the per- 
centage of students in the high-risk subgroup who completed a basic core academic curriculum 
(four English courses, three social studies courses, two math courses, and two science courses). 
At the same time, students in the Academy group were significantly more likely than their non- 
Academy counterparts to complete three or more career-related or vocational courses. 

The fifth set of bars in Figure ES-1 indicates that the Academies increased the percentage 
of students in the high-risk subgroup who reported that they had submitted an application to a 
two-year or four-year college by the end of their ^“’-grade year. In particular, 35 percent of stu- 
dents in the high-risk non-Academy group reported submitting a college application, compared 
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Figure ES-1 

Career Academy Impacts on High School Outcomes 
in the High-Risk Subgroup 
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SOURCES: MDRC calculations from the Career Academies Evaluation Student School Records and 12th Grade 
Survey Databases. 



NOTE: A two-tailed t-test was applied to differences between Academy and non-Academy groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 



with 51 percent in the Academy group. Academy students in the high-risk subgroup were also 
more likely to report taking the SATs or ACTs (not shown in the figure). 

Finally, the last two sets of bars in Figure ES-1 indicate that Academies did not produce a 
systematic change in involvement in positive youth development activities or in negative risk- 
taking behaviors. Positive youth development activities included participation in community 
volunteer work, receiving recognition for participation in academic or extracurricular activities, 
and receiving an academic award. Negative risk-taking behaviors included coming to school on 
drugs, becoming a parent, being expelled from school, and being arrested. Although the differ- 
ences between the groups shown in Figure ES-1 were not statistically significant, they indicate 
trends in a positive direction. 



Career Academy Impacts for Students in the Low-Risk Subgroup 

Figure ES-2 presents a summary of the impact findings for students in the low-risk sub- 
group. The results for the non-Academy group indicate that, even without access to the Academy 
intervention, these students appear to be unlikely to disengage from school. For example, as the 
first set of bars in Figure ES-2 illustrates, only 3 percent of the non-Academy students in the 
low-risk subgroup dropped out of high school before the end of 12 th grade. Almost the same per- 
centage of Academy students (2 percent) dropped out. 
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Figure ES-2 

Career Academy Impacts on High School Outcomes 
in the Low-Risk Subgroup 




SOURCES: MDRC calculations from the Career Academies Evaluation Student School Records and 12th 
Grade Survey Databases. 



NOTE: A two-tailed t-test was applied to differences between Academy and non-Academy groups. Statistical 
significance levels are indicated as: *** = 1 percent; * * = 5 percent; * = 10 percent. 



o Career Academies increased the likelihood that students in the low-risk 
subgroup were prepared to graduate on time. For these students, the 
Academies also increased career-related and vocational course-taking 
without reducing the likelihood off completing a basic academic core cur- 
riculum. 

The second set of bars in Figure ES-2 indicates that the Academies increased the percent- 
age of students in the low-risk subgroup who earned sufficient credits to meet their district’s 
graduation requirement. The figure shows that 86 percent of the Academy students met their dis- 
tricts’ graduation requirement, compared with 75 percent of the students in the non-Academy 
group. 



Also, while approximately equal percentages of Academy and non-Academy students in 
the low-risk subgroup completed a basic core academic curriculum, the Academies significantly 
increased the percentage who completed at least three career-related or vocational courses. It 
should be noted than many students in the low-risk non-Academy group were likely to be en- 
rolled in their high school’s college preparatory programs and courses. Thus, the Academies in- 
creased vocational course-taking for the low-risk subgroup while enabling students to complete 
as many core academic courses as their non-Academy peers. 

The fifth set of bars in Figure ES-2 indicates that the Academies reduced the percentage 
of the low-risk subgroup who reported that they had submitted an application to a two-year or 
four-year college by the end of their ^^-grade year. Among these students, 79 percent of the 
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non-Academy group reported submitting a college application, compared with 71 percent of the 
Academy group. Although not shown in the figure, this occurred despite the fact that Academy 
and non-Academy students were equally likely to have taken the SATs and ACTs. In addition, 
over 85 percent of students in both low-risk groups reported that they had conducted at least a 
modest amount of research on college options during their ^‘’’-grade year. 

Figure ES-2 also shows that Academy and non-Academy students in the low-risk sub- 
group were equally likely to pursue post-secondary employment opportunities. Further analyses 
indicated that the Academies do not appear to have induced students to pursue post-secondary 
employment opportunities instead of either a two-year or four-year college. Further follow-up is 
needed to determine the effects that the Career Academies may have had on actual college en- 
rollment and employment during the years following high school graduation. This will be ex- 
plored further in subsequent reports from the Career Academies Evaluation. 

Finally, the last two sets of bars in Figure ES-2 show that the Academies did not produce 
statistically significant changes in the low-risk subgroup’s involvement in positive youth devel- 
opment activities or risk-taking behavior. 



Career Academy Impacts for Students in the Medium-Risk Subgroup 

• On average, the Career Academies produced little or no change in out- 
comes for students in the medium-risk subgroup. Results for medium- 
risk students differed considerably across the participating sites. 

The medium-risk subgroup represents approximately 50 percent of the students in the 
study sample. As shown in Table ES-1, the characteristics of this subgroup do not provide a clear 
indication of likely school success or disengagement. Figure ES-3 presents a summary of impact 
findings for students in the medium-risk subgroup. The figure indicates that, on average, the 
Academies had little or no impact on most outcomes for these students. As discussed below, 
however, the results for the medium-risk subgroup differed dramatically across the participating 
sites. 



Impact Findings for the Full Sample 

• When averaged across the diverse groups of students and sites partici- 
pating in the evaluation, it appears that the Career Academies produced 
only modest improvements in students’ engagement and performance 
during high school. 

Figure ES-4 provides a summary of impact findings that are averaged across the full 
sample of students in the study. It suggests that the Academies produced only slight (and not sta- 
tistically significant) reductions in dropout rates and in student involvement in negative risk- 
taking behaviors. On average, the Academies produced modest increases in the percentage of 
students who earned sufficient credits to meet district graduation requirements and in student in- 
volvement in youth development activities. In keeping with one of the central features of the 
Academy approach, Figure ES-4 indicates a more substantial increase in vocational course- 
taking. This increase did not come at the expense of students’ being less likely to complete at 
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Figure ES-3 

Career Academy Impacts on High School Outcomes 
in the Medium-Risk Subgroup 
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SOURCES: MDRC calculations from the Career Academies Evaluation Student School Records and 12th Grade 
Survey Databases. 

NOTE: A two-tailed t-test was applied to differences between Academy and non-Academy groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** ~ 5 percent; * = 10 percent. 



least a basic core academic curriculum. In general, however, according to the full sample find- 
ings, the Career Academies tended to produce small, positive (but not statistically significant) 
impacts on many student outcomes. As discussed earlier, these aggregate findings mask a great 
deal of underlying variation that sheds light on the potential strengths and limitations of the 
Academy approach. 

• The Career Academies did not improve standardized measures of reading 
and math achievement either on average or for any subgroup of students. 

According to standardized achievement tests completed by 490 students in the study 
sample, the Career Academies did not produce any systematic improvement in students’ math 
and reading test scores. Although impacts on test scores followed trends found for other out- 
comes, such as academic course-taking, there was no clear pattern of increases or decreases ei- 
ther on average or among the risk subgroups. 

Among students in the high-risk subgroup, average math and reading test scores for the 
Academy group were somewhat higher than scores for the non-Academy group. While none of 
the differences was statistically significant, test scores followed this subgroup’s trend of in- 
creases in academic course-taking and total credits earned toward graduation. Academy students 
in the low- and medium-risk subgroups had slightly lower reading test scores than their non- 
Academy counterparts. This is consistent with the slight (but not statistically significant) reduc- 
tion in academic course-taking, which was found to be more highly correlated with reading test 
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scores than was non-academic course-taking. There was almost no difference in math test scores 
between Academy and non-Academy students in the low- and medium-risk subgroups. 

Several factors may account for these test score findings. First, qualitative field research 
information collected for this evaluation indicated that academic curricula and instruction in most 
of the Career Academies did not differ substantially from those of typical high schools; Academy 
teachers were required to cover the same basic material as teachers of the same subjects in the 
rest of the high school. Nor were Academy teachers typically provided with professional devel- 
opment opportunities beyond those offered to their non-Academy counterparts, which focused on 
standard-setting and instructional strategies in the academic subjects. 

Second, there were some important differences between the sample of students who com- 
pleted the math and reading achievement tests and those who did not. In particular, the magni- 
tude of impacts for the achievement test sample was somewhat smaller and more mixed than the 
magnitude of impacts described above. For example, among students in the high-risk subgroup 
who completed the math and reading tests, the Academies produced a somewhat smaller reduc- 
tion in dropout rates and a somewhat smaller increase in academic course-taking compared with 
the impacts displayed in Figure ES-1. Among students in the medium-risk subgroup who com- 
pleted the test, it appears that the Academies actually reduced academic course-taking. In short, 
the test score sample does not appear to be representative of the full study sample. Nonetheless, 
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there was not a systematic difference in background characteristics between the Academy and 
non-Academy students in the achievement test sample. Thus, test score impact estimates provide 
a reliable indication of the Academies’ impact (or lack of impact) on test scores. 

Finally, the types of standardized measures of achievement used in this evaluation, and in 
many school districts, may not adequately capture learning gains that Academy students achieve 
relative to their non-Academy counterparts. As discussed in an earlier report from this evalua- 
tion, Academy teachers were more likely than their non-Academy colleagues to state that they 
made explicit efforts to plan lessons and activities that cut across academic and non-academic 
subject areas. 7 They were also more likely to have students focus on problem-solving activities 
and to integrate problems and examples from the world of work into their lessons. Academy stu- 
dents were more likely than their non-Academy peers to indicate that they received instruction 
that included cross-discipline integration and connections between school-based and work-based 
learning. If the potential benefits of such activities and experiences are of value to schools, they 
will likely need to be measured through some alternative forms of assessment. 



What Factors Help Explain the Pattern of Career Academy Effects? 

Figure ES-5 illustrates a conceptual model of the pathways through which the core or- 
ganizational features of the Career Academy approach are hypothesized to affect student out- 
comes during high school and beyond. The first column of the figure lists the three core organiza- 
tional elements of the Career Academy approach: (1) the school-within-a-school, (2) the integrated 
academic and vocational curricula based on the Academy’s career theme, and (3) the employer 
partnerships. Three types of supports and learning opportunities (the second column in Figure ES- 
5) are hypothesized to evolve from the core organizational elements and their interaction: (1) en- 
hanced interpersonal support through the intensive collaboration offered by the school-within-a- 
school, (2) focused curricula and enriched teaching and learning through the combination of aca- 
demic and vocational courses, and (3) exposure to career awareness and work-based learning op- 
portunities through the employer partnerships. Together, these supports are intended to increase 
students’ school engagement and prevent them from dropping out, enhance their performance and 
help them meet graduation requirements and prepare for post-secondary education and employ- 
ment, and promote constructive use of non-school hours by increasing developmentally appropriate 
activities and reducing risk-taking behaviors. 

For this report, a variety of analyses were aimed at assessing the relationships between stu- 
dent outcomes and measures of supports and learning opportunities that are likely to arise from the 
Career Academy’s organizational elements. The findings from these analyses suggest that the 
strongest associations appear to exist between the interpersonal supports students received early in 
high school and various measures of their subsequent engagement and performance. The interper- 
sonal supports include students’ perceptions of their teachers’ expectations for them, personalized 
attention they receive from teachers, the degree to which they see their peers as being engaged in 
school, and the degree to which they have opportunities to work collaboratively with peers. 



7 James J. Kemple, “Selected Dimensions,” cited above. 
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Both Academy and non-Academy students who reported that they received particularly 
high levels of support from their teachers and peers in 9 th or 1 0 th grade were less likely to drop out 
of high school, exhibit chronic absenteeism, or engage in risk-taking behaviors than were students 
who reported lower levels of interpersonal support. They were also more likely to make steady pro- 
gress toward graduation and to engage in positive youth development activities. One should be 
cautious about making inferences about causal relationships in this regard. For example, students 
who achieve positive outcomes may attract strong support from teachers and peers, rather than the 
other way around. Nevertheless, the patterns of cross-site impacts described below provide further 
evidence that interpersonal supports are likely to be important antecedents to positive outcomes for 
students. 
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o In several participating sites, the Career Academies represented a par- 
ticularly dramatic contrast with their non-Academy school environments. 
Specifically, these Academies produced particularly large increases in the 
level of interpersonal support students received early in high school, rela- 
tive to the level experienced by students in the non-Academy environ- 
ments. 

To explore the relationship between changes in the school environment that the Acade- 
mies represent and the impact that Academies have on student outcomes, the evaluation at- 
tempted to identify sites in which Academies produced the largest differences in the level of in- 
terpersonal support students experienced. Specifically, the individual sites in the evaluation were 
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ranked according to the difference between the percentages of Academy and non- Academy stu- 
dents who reported receiving a high level of support from teachers and peers during 9 th or 10 th 
grade. For the purposes of this report, the sites with the largest differences are referred to as high- 
contrast Academies. In the remaining sites, there was little difference in the level of support re- 
ported by Academy and non-Academy students; these sites are referred to as low-contrast 
Academies. 

Finally, there are several important similarities between the two groups of sites. Both 
high-contrast and low-contrast Academies produced substantial increases in students’ exposure 
to career awareness and development opportunities and their participation in work-based learning 
activities. It should be noted, however, that within the two groups of sites, some Academies pro- 
duced much larger increases in students’ exposure to these activities and experiences than others. 

• The high-contrast Academies produced a consistent pattern of positive 
impacts for students in the medium-risk subgroup. On average, the low- 
contrast Academies increased dropout rates and reduced academic 
course-taking among these students. 

The patterns of impacts for students in the medium-risk subgroup differed dramatically 
between the high-contrast Academies and the low-contrast Academies. As shown in Figure ES-6, 
in general the high-contrast Academies produced impacts that were similar but smaller in mag- 
nitude to impacts for students in the high-risk subgroup (Figure ES-1): they reduced dropout 
rates, increased credits earned toward graduation, and increased the percentage of students com- 
pleting a basic core academic curriculum. Figure ES-6 also indicates that the low-contrast 
Academies actually increased dropout rates and reduced the percentage of students who com- 
pleted a basic core curriculum. * 

While it is not possible to pinpoint the source of differences in impact findings for high- and 
low-contrast Academies, differences in program implementation may suggest some explanations. 
For example, qualitative field research information collected for the evaluation indicated that the 
high-contrast Academies tended to have implemented a tighter school-within-a-school organization 
compared with the low-contrast sites. The high-contrast Academies typically included a core group 
of four or five teachers whose responsibilities fell almost exclusively within the Academy. The vast 
majority of students in high-contrast sites were scheduled together in at least two or three core 
courses, and very few non-Academy students had to be included in the Academy classes (for ex- 
ample, to ensure adequate enrollments). The high-contrast Academies also tended to be located in a 
distinct area of the school building or campus. These features of the high-contrast Academies may 
have nurtured a more personalized learning environment and helped students and teachers feel that 
they were part of something unique within the school. The tightly organized school-within-a-school 
may also have served as a foundation for enhancing instructional supports, curriculum integration, 
and connections between school and work. 

The school-within-a-school organization of the low-contrast Academies tended to be more 
loosely structured and typically included several teachers who had responsibilities both in and out- 
side the Academy. A number of Academy students in low-contrast sites were scheduled in non- 
Academy sections of core courses, and several of the Academy classes included non-Academy stu- 
dents in order to ensure adequate enrollments. These aspects of program implementation tended to 
minimize the contrast between the Academy and non-Academy environments. It is difficult to de- 
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termine how this might account for the apparent reduction in school engagement among the me- 
dium-risk subgroup in these sites. It may be that without a tightly organized, highly supportive 
school-within-a-school environment, the other aspects of the Academy experience (additional vo- 
cational courses, career awareness activities, and work-based learning) may have become somewhat 
of a distraction or burden. 



Figure ES-6 
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In general, the patterns of impacts for the high-risk and low-risk subgroups were consis- 
tent across both groups of sites, with two notable exceptions. First, the low-contrast Academies 
produced a somewhat larger reduction in dropout rates among the high-risk subgroup. Although 
the difference in impacts on dropout rates was not statistically significant, this pattern is not con- 
sistent with the hypothesis that greater enhancement of interpersonal supports should lead to 
larger reductions in dropout rates. It is not clear what accounts for the pattern. Second, the low- 
contrast Academies produced somewhat larger increases in vocational course-taking for both the 
high-risk and the low-risk subgroups. This may reflect a greater emphasis on vocational course- 
taking in low-contrast sites and the fact that, on average, relatively few non-Academy students in 
these sites completed three or more career-related or vocational courses during high school. 



Policy ImpflkattMms amd Lessons for Practice 

Although the story of the Career Academies’ longer-term effectiveness is not yet com- 
plete, the findings to date suggest the following implications and lessons. 

° The Career Academies in this study demonstrate the feasibility of imple- 
menting a well-defined and effective approach to creating a more suppor- 
tive high school environment and increasing students’ exposure to career 
awareness and work-based learning activities. 

Large comprehensive high schools (including those participating in this study) have been 
criticized for being impersonal and for preventing students and teachers from working as teams 
to create a sense of community and common values. Students in such schools do not have a con- 
sistent group of teachers who are accountable for their success, and they see few of the same 
classmates from course to course. Teachers rarely share the same group of students with a small 
number of colleagues, and they have few opportunities to coordinate their coursework with 
teachers in other disciplines. The findings from this evaluation provide evidence that the Career 
Academies can provide well-defined and effective approaches to addressing such problems. 

Another common problem identified in high schools is that students and teachers are iso- 
lated from other institutions in the community, particularly employers. Such isolation insulates 
students from the world of work and misses an opportunity to provide them with learning- 
oriented exposure to it at a particularly formative point in their development. With few connec- 
tions among classes or between school and work, many students are inadequately informed about 
or are unprepared for post-secondary education and employment opportunities. Even with the 
rise of the school-to-work movement and with the federal School-to-Work Opportunities Act of 
1994, there has been a struggle to identify widely implemented strategies that address these con- 
cerns. The findings from this evaluation indicate that Career Academies can provide concrete 
examples of partnerships between schools and employers and can substantially enhance students’ 
exposure to career development and work-based learning opportunities. 

© Career Academies are an effective means of enhancing the school en- 
gagement of students who are at high risk of dropping out of high school. 

Many of the students served by Career Academies enter high school at a substantial risk 
of dropping out; many others are likely to become psychologically disengaged from school and 
to make only limited progress toward graduation. Some of these students have already fallen be- 
hind or are disengaged when they enter high school, while others come from home environments 
that lack the support or resources to facilitate academic persistence and success. Without the in- 
tervention of the Academies, about 1 in 3 of these young people will drop out of high school. 
Previous research has shown that the economic and social costs of not securing a high school di- 
ploma are extremely high. 

The findings from this evaluation show that the Career Academies substantially reduced 
dropout rates and substantially improved a variety of measures of school engagement among stu- 
dents in the high-risk subgroup. Not only are effects of this magnitude and pervasiveness rare in 
the world of education policy interventions, but the long-term payoff, if the effects persist, is 
likely to be large. 
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o Career Academies should comtiuue to serve a heterogeneous student 
population. 

Because the largest and most pervasive positive effects in this evaluation were found 
among students in the high-risk subgroup, it might be argued that the Career Academies should 
serve only such students. This approach is likely to create a number of problems, however. First, 
Career Academies have explicitly attempted to move away from targeting students on the basis 
of their estimated trajectories for school success in order to avoid the tracking and stigma that 
have been associated with vocational and career-related programs. Second, and perhaps more 
important, it is likely that exposure to a broad cross-section of students — particularly those who 
enter the programs highly engaged in school — is an important factor driving the positive effects 
of Career Academies on the high-risk subgroup. Perhaps the presence of other, highly engaged 
students in their classrooms helps increase teachers’ attention to and expectations for all students. 
Excluding engaged students, therefore, would dramatically change the nature of the Academy 
experience for students at high risk of dropping out. 

o Interpersonal supports appear to be necessary conditions for maximizing 
the positive effects Career Academies have on student engagement. The 
school-within-a-school organization can provide an effective strategy for 
enhancing these supports. 

The findings indicate that enhancing interpersonal supports may be a key element of 
school reform initiatives aimed at increasing retention and engagement in school. A highly 
structured school-within-a-school organization can provide some of the necessary conditions for 
promoting such supports as personalized attention and high expectations from teachers, high lev- 
els of peer engagement, and opportunities for teachers and students to work collaboratively. Ca- 
reer Academies that did not complement their career awareness and work-based learning activi- 
ties with increased supports (relative to what was already available in the regular school envi- 
ronment) risked having some of their students become disengaged from school. 

o Although Career Academies provide a number of supports necessary to 
keep students engaged in school, these have not been sufficient to enhance 
achievement, at least as measured by commonly used standardized tests. 

The primary added value of Career Academies appears to rest on their enhanced interper- 
sonal supports and increased access to career awareness and work-based learning opportunities. 
Although these factors may be necessary to keep many students engaged in school, they were not 
sufficient to improve student achievement. The findings from this evaluation indicate that the 
Career Academies were quite similar to regular school environments in terms of their academic 
curricula and typical instructional strategies. From this standpoint, it should not be too surprising 
that the Academies did not I mprove student achievement as measured by the standardized math 
and reading tests used in the evaluation. Academies face many of the same challenges that most 
high schools do in providing teachers and students with appropriate incentives and supports to 
ensure that they focus on clearly specified learning objectives and that they have the capacity to 
meet those objectives. The personalized and collaborative nature of the Career Academy’s 
school-within-a-school organization can serve as a solid foundation on which to build these en- 
hancements. 
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There is also a question about whether current assessment instruments (including the 
achievement tests used in this evaluation) adequately capture the distinctive learning gains that 
Academy students may attain. Such skills may include, for example, the type of work-related 
competencies outlined in the Secretary’s Commission on Achieving Necessary Skills (SCANS) 
or the presentation and organizational skills often exhibited in student portfolio assessments. In 
order to measure such potential benefits of a Career Academy, school officials may want to con- 
sider incorporating alternative forms of student and teacher assessment. They may also want to 
develop forums that recognize efforts by teachers to integrate academic course content with the 
applied learning and problem-solving approaches of high-quality vocational curricula. Few ex- 
amples of such assessments and incentives currently exist. 

• Longer-term follow-up is needed to ascertain the effects of Career 
Academies on post-secondary labor market and educational outcomes. 

The results in the report summarize the effects that the Career Academies have had on 
students through the end of the year they were scheduled to be in 12 th grade. The data do not in- 
clude complete information about actual high school graduation rates or about the critical transi- 
tion between high school and post-secondary education and work. Further follow-up is needed in 
order to get a more complete picture of the Academies’ effectiveness and limitations. For exam- 
ple, it will be important to determine whether the reduction in dropout rates among students in 
the high-risk subgroup translates into higher levels of educational attainment or whether these 
students simply remain in school longer without earning a diploma or do not go beyond high 
school. It will also be important to determine whether the Career Academy experience helped or 
hindered students in the low-risk subgroup, particularly regarding their actual rates of college 
enrollment and completion. Ultimately, measures of success for Career Academies are likely to 
depend, in part, on whether the students they attempted to serve are better attached and more 
successful in the labor market than their non- Academy counterparts. 

In order to examine these issues, MDRC’s Career Academies Evaluation will continue 
through 2003, following the students in the study sample for up to four years after their sched- 
uled graduation from high school. As part of this second phase of the evaluation, MDRC will be 
administering follow-up surveys to students in the study sample at one year and four years fol- 
lowing their scheduled graduation. These surveys will provide information about whether the 
students’ actually graduated from high school (or received an alternative credential) and about 
their enrollment and progress through post-secondary education, their labor market experiences, 
their preparation and planning for the future, and a range of youth development experiences. 
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Chapter 1 

Introduction 



This report summarizes results from the first phase of the Career Academies Evaluation 
being conducted by the Manpower Demonstration Research Corporation (MDRC). Career 
Academies have existed for over 30 years and have been implemented in approximately 1 ,500 
high schools across the country. The durability and broad appeal of the Academy approach can 
be attributed, in part, to the fact that its core features offer direct responses to a number of prob- 
lems that have been identified in large comprehensive high schools. 

Career Academies attempt to create more supportive and personalized learning environ- 
ments through a school-within-a-school structure. Their curricula combine academic and occu- 
pation-related course requirements that aim both to promote applied learning and to satisfy col- 
lege entrance requirements. Academies establish partnerships with local employers to build se- 
quences of career awareness and work-based learning opportunities for their students. The pri- 
mary goals of the Career Academy approach are to enhance students’ performance in high school 
and provide them with the credentials and skills needed to make a successful transition to post- 
secondary education and, eventually, a career. 

This is the latest in a series of reports and papers from MDRC’s Career Academies 
Evaluation, which is being funded by the U.S. Departments of Education and Labor and 17 pri- 
vate foundations and organizations. It adds to findings presented earlier in this evaluation in sev- 
eral ways. 

First, this report assesses the impact Career Academies have on students’ high school en- 
gagement and performance and on their preparation for post-secondary education and employ- 
ment. The previous reports from this study presented descriptive information about the Academy 
programs and focused on measures of students’ exposure to key dimensions of the Academy ap- 
proach earlier in their high school careers. This report adds to these findings by following stu- 
dents in the study sample through the end of their ^^-grade year and by examining a much more 
extensive set of student outcomes. It assesses the impact Career Academies have on keeping stu- 
dents enrolled and engaged in school, on the types of courses they take, on math and reading 
achievement test scores, on participation in extracurricular activities, on risk-taking behaviors, 
and on whether students are prepared to enter post-secondary education and employment. As dis- 
cussed later in this chapter, the evaluation is built on a random assignment research design that 
can provide unusually rigorous evidence about the impact Career Academies have on students. 

Second, this report examines the relative effectiveness of the Academy approach for sev- 
eral key subgroups of students and among the sites represented in the study sample. The previous 
reports and papers from MDRC’s Career Academies Evaluation focused primarily on findings that 
were aggregated across the full sample of students and sites in the study sample. While such find- 
ings shed light on the implementation and impact of the Career Academy approach more generally, 
they mask the extent to which the Career Academies may change certain outcomes for some stu- 
dents but not necessarily for others. They also mask the high degree of variation among the sites 
and the ways this variation may be associated with differences in effectiveness. A central goal of 
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this report, therefore, is to determine how the manner and context in which Career Academies are 
implemented affect their capacity to make a difference for students. 

Third, this report attempts to push much further in offering lessons about the efficacy of 
the Career Academy approach and of other school reform initiatives that are embedded in it. 
While the findings presented here provide an assessment of the effectiveness of particular Career 
Academies, the study design and the available data provide unique opportunities to go beyond 
this. For example, the sites vary significantly in terms of the types of interpersonal and instruc- 
tional supports they offer students and in the types of work-related learning opportunities they 
provide. The contrast among the sites and the differences in effectiveness for key subgroups of 
students provide a rich context for making judgments about what types of school reform initia- 
tives are likely to be effective and for whom. 

Finally, this report serves as a platform for further analyses to determine the long-term 
impact of the Career Academy approach. MDRC’s Career Academies Evaluation is scheduled to 
continue until 2003, following students for up to four years beyond high school. During this sec- 
ond phase of the evaluation, MDRC will prepare additional reports examining the effect Career 
Academies have on students’ post-secondary outcomes and exploring connections between high 
school experiences and success in higher education and the labor market. 

The remainder of this introductory chapter sets the context for the Career Academies 
Evaluation and this report. It is divided into four sections. Section I presents a short history of 
the Career Academy approach and sets the current policy context for the evaluation and its 
findings. Section II reviews findings from previous research on Career Academies and high- 
lights areas where the current evaluation can fill important gaps in what is known about their 
effectiveness. Section III describes a conceptual framework for understanding the Career 
Academy approach and its potential effects on students during high school and beyond. Sec- 
tion IV describes several key features of the Career Academies Evaluation design that are par- 
ticularly relevant to this report. 



I. Tjhie Origms of tjhie Career Academy Approach and the Policy Context 
for TMs Report 

The interpretation and significance of the findings from this evaluation should be viewed 
in the context of both the history of the Career Academy approach and the current policy initia- 
tives that intersect with that history. This section of the chapter provides a brief summary of the 
origins of the Career Academy approach and an overview of the policy context into which this 
report will be released. 

A. The Origins and Growth of tine Career Academy Approach 

The first Career Academy was established in 1969 in Philadelphia, Pennsylvania.' This 
Electrical Academy was designed primarily as a vocational training program targeted to non- 



'For a more detailed history of the Career Academies, see Stem, Raby, and Dayton, 1992; Academy for Educa- 
tional Development, 1989; Snyder and McMullan, 1987. The term Career Academy was designated by Stem, Raby, 
and Dayton to encompass all the various strands of academies that had evolved up to that point. 
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college-bound students. Building on the Philadelphia experience, in the 1980s, the Edna McCon- 
nell Clark Foundation provided initial funding to establish Academies in Pittsburgh, Pennsylva- 
nia; Portland, Oregon; and Menlo-Atherton and Redwood City, California. Based on the experi- 
ence of the programs established in Menlo-Atherton and Redwood City (known as the Peninsula 
Academies), the California State Legislature passed a bill providing funding for up to 10 school 
districts to establish new Academies (later referred to as California Partnership Academies) be- 
ginning in the 1985-86 school year. Meanwhile, the American Express Company, in collabora- 
tion with the New York City Public schools, established Academy programs focused on the fi- 
nancial industry. By the end of the 1980s, it is estimated that there were over 100 Academies in 
Philadelphia, California, and the cities that received Clark Foundation start-up grants or support 
from the American Express Company. 

These early Career Academies shared several characteristics that have important implica- 
tions for the current state of Academies and the Academy movement. First, the initial Academies 
were primarily vocational education programs targeted for students who appeared to be at high 
risk of dropping out of high school. The central goals of these early programs were to keep stu- 
dents engaged in school, provide them with work-related learning experiences both in the class- 
room and on the job, and establish clearer pathways between high school and work. This vision 
for the Career Academy approach was adopted, in large part, to deal with many of the major 
problems that were identified with American high schools in the 1970s and 1980s. In fact, the 
legislation authorizing funding for the California Partnership Academies expressly states that the 
programs must target “educationally disadvantaged high school students,” defined as “students 
who are at risk of dropping out of high school.” 2 

Second, the early programs in Philadelphia and California established the basic organiza- 
tional elements that eventually came to define the Career Academy approach as a distinctive high 
school reform initiative. They were organized as schools-within-schools and used a career theme 
to help coordinate academic and vocational curricula. Each of the programs also established 
partnerships with local employers to build connections between school and work for its students, 
and to secure funding for the programs. Although these organizational features were not used ex- 
plicitly to define the Career Academy approach until recently, they were clearly identifiable and 
could be replicated under a wide range of circumstances. 3 

Third, beginning with the very first Academies in Philadelphia, there have been efforts to 
document their success and to justify their ongoing operation and expansion on the basis of their 
evidence. The initial replications sponsored by the Edna McConnell Clark Foundation were un- 
dertaken, in part, because of the documented success of the Philadelphia Academies. The Clark 
Foundation, along with the William and Flora Hewlett Foundation, also provided funding for an 
evaluation of the first 10 California Academies. The evidence from this evaluation played a key 
role in extending and expanding the state legislation to create many more Academies. In addition 
to evaluation research, the initial Academies were the subject of implementation research to 
document strategies for creating and sustaining new programs. This research and documentation 



2 Stem, Raby, and Dayton, 1992. 

’Researchers, policy advisors, and practitioners at the Career Academy Support Network engaged in an exten- 
sive consensus-building process among various organizations and individuals associated with Career Academies to 
arrive at a commonly agreed-upon definition of a Career Academy. This definition is articulated in Stem, Dayton, 
and Raby, 1998. 
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led to more systematic technical assistance and staff development efforts for both existing 
Academies and those in the planning stages. 

B. The Current State of the Career Academy Movement 

The 1990s have seen extraordinary growth in the number of Career Academies around the 
country. Currently, there are estimated to be approximately 1,500 Career Academies nationwide 
(nearly a 15-fold increase over 10 years) and many more in the planning stages. Much of this 
growth can be traced to the increasing number of national, state, and district Academy support 
networks. As of the 1998-99 school year, the California Department of Education has provided 
funding for nearly 200 Partnership Academies. It also provides support for several technical as- 
sistance and professional development services, including an annual conference. In addition to 
the state-funded Partnership Academies, there are estimated to be over 200 other Academies in 
California that are patterned after the Partnership Academy model but are supported through lo- 
cal efforts. Building on the California legislation and Partnership Academy model, Illinois, 
Florida, Hawaii, and other states have also established statewide networks of Academies. In 
1988, a consortium of business, labor, and education leaders established the Philadephia High 
School Academies (PHSA), Inc., to oversee the network of 28 Academies in Philadelphia. Since 
then, a growing number of other cities have developed Academy networks, including Atlanta, 
Baltimore, Chicago, Denver, Oakland, Pasadena, Seattle, and Washington, DC. 

In 1988, the American Express Company and other employer partners established the 
National Academy Foundation (NAF) to coordinate the expansion and ongoing development of 
the Academy of Finance model. Since then, NAF has received its largest support from American 
Express and Citigroup and has grown to include nearly 400 Academies nationwide. The NAF 
model has also been expanded to include themes in travel and tourism and public service. Re- 
cently, NAF began work on an Information Technology Academy model. 

More recently, there have been other initiatives to create national Academy support net- 
works, and these networks have begun working together to coordinate their efforts. In 1996, the 
National Career Academy Coalition (NCAC) was established by a consortium of technical as- 
sistance providers including PHSA, Inc., and GMS Partners, which had been providing technical 
assistance to Academies in Washington, DC. NCAC received endorsements from several federal 
agencies led by the U.S. Department of the Treasury, and it offers an annual technical assistance 
conference for new and established Academies nationwide. In 1998, the Dewitt Wallace- 
Reader’s Digest Fund provided funding to establish the Career Academy Support Network 
(CASN) based at the University of California at Berkeley. CASN had led an effort to build con- 
sensus for a definition of a Career Academy and has developed a range of technical assistance 
tools for states, school districts, and schools interested in creating new Academies. 

In addition to growth in the number of Academies, there has been a shift in the primary 
goals and target population of most Career Academies. In particular, there is now wide agree- 
ment that the Career Academy approach should be explicitly distinct from traditional vocational 
education by seeking to prepare students for both work and college. 4 Vocational education, as 
defined in federal law and through its historical legacy, has been directed at preparing young 



4 See Stem, Dayton, and Raby, 1998, for a discussion of the current definition of a Career Academy and its key 
goals. 
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people for occupations that do not traditionally require advanced degrees. By contrast, Career 
Academies now seek to include a broad range of students and to combine a rigorous academic 
curriculum with exposure to extensive information about an industry, both in the workplace and 
in the classroom. 

Finally, although most Career Academies today continue to share the approach’s basic 
organizational elements, the Academy model has been adapted to a wide range of local needs and 
circumstances, resulting in different versions of the approach that emphasize some features over 
others. For example, the NAF network has focused a great deal on creating theme-related curric- 
ula in finance and travel and tourism, while many of the California programs placed somewhat 
greater emphasis on creating smaller learning communities through the school-within-a-school 
structure. Both types of programs have sought to develop strong employer partnerships. Even 
within the California, Philadelphia, and NAF expansion efforts there has been considerable 
variation in the roles employers played, the strategies used to integrate academic and vocational 
curricula, and the particular characteristics of their school-with-a-school organization. This 
variation highlights the adaptability of the Academy approach. 

The expansion of the target populations and goals of the Career Academy approach, as 
well as the rapid growth in the number of Academies, has placed a premium on obtaining high- 
quality information about how the Academies may be affecting students’ performance in high 
school. How well does the Academy approach fit the needs of college-bound students, as well as 
those who may be at high risk of dropping out? Are Academies more effective under some con- 
ditions than under others? Which aspects of the Academy approach make the most difference for 
students? MDRC’s evaluation is intended to shed valuable light on these and other questions. 

C. Tine Current Policy Context 

This report is being released at a time when education policymakers and practitioners 
are pursuing a number of far-reaching strategies for improving American high schools. In ad- 
dition to the rapid expansion of Career Academies and Academy support networks, there have 
been several policy and school reform initiatives that build on or directly incorporate the Ca- 
reer Academy approach. Most notably, the School-to-Work Opportunities Act (STWOA) of 
1994 was designed to catalyze fundamental changes in the way states and localities support 
partnerships between employers and schools. These partnerships have been aimed at enhancing 
the relevance and rigor of school- and work-based learning and at creating clearer pathways 
between high school and post-secondary education and careers. STWOA specifically identifies 
Career Academies as a “preferred approach” to creating such partnerships and implementing 
the principles embedded in the legislation. States, school districts, and employers are now 
looking for strategies and approaches that can build on the foundation established by STWOA 
and address some of its limitations. 

Federal education policymakers have been considering significant changes to the Ele- 
mentary and Secondary Education Act (ESEA). Such changes are likely to build on proposals 
that have already been put forward to address problems unique to high schools. For example, the 
Educational Excellence for All Children Act of 1999 supports education reforms in 5,000 Ameri- 
can high schools that will aid students by improving schoolwide Title I school programs, 
strengthening curricula and instruction and providing better professional development opportu- 
nities for school staff. The act emphasizes the need for creating smaller learning environments, 
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involving members of the community in schools, and establishing partnerships with other insti- 
tutions as important ways to promote safer and more supportive schools. These are all elements 
that are present in the Career Academy approach. 

The Office of Vocational and Adult Education (OVAE) has established the New Ameri- 
can High Schools Initiative to showcase high schools that have implemented a diverse set of 
strategies for education reform, particularly focused on preparation for college and work. The 
initiative has provided evidence that high school reforms that are locally driven and standards- 
based are associated with better attendance and graduation rates for students. 

Other researchers and reformers have also been developing a variety of approaches to im- 
proving high schools. These reforms include High Schools That Work (HSTW), developed by 
the Southern Regional Education Board. HSTW is a whole-school, research-based reform de- 
signed to raise the academic achievement of career-bound high school students by combining the 
traditional college preparatory curriculum with vocational classes. The Talent Development 
Model High School, developed at the Center for Research on the Education of Students Placed at 
Risk (CRESPAR), specifically includes Career Academies as a central feature of its reform ap- 
proach. The program divides large, urban high schools into smaller learning communities: a 9th 
Grade Success Academy and Career Academies in grades 10 through 12. Project Graduation 
Really Achieves Dreams (GRAD) was developed by the Houston Public Schools and uses a 
combination of innovative programs to build students’ interpersonal and academic skills. Project 
GRAD begins in elementary and middle schools and then, in high schools, seeks to implement 
personal mentoring relationships and financial incentives for college. Currently, Project GRAD 
is considering implementing a series of Career Academies within secondary schools to ensure 
that the gains made in elementary and middle school are sustained and enhanced by small learn- 
ing communities and integrated curricula. Finally, the Institute for Research and Reform in Edu- 
cation (IRRE) has developed the First Things First (FTF) initiative in Kansas City, Kansas, to 
help improve feeder systems of elementary, middle, and high schools. The cornerstones of FTF 
are consistent with key features of the Academy approach, including efforts to create small 
learning communities; build strong relationships among students, parents, and teachers; and fos- 
ter collaborative and active learning opportunities based on academic standards. 

Virtually all these approaches to improving American high schools include principles 
embedded in the Career Academy approach and, in some cases, include the Career Academy 
model as an explicit component. The findings presented in this report will shed light on the ex- 
tent to which the Career Academy model, and the key reform approaches embedded in the 
model, can achieve the goals that have been espoused by their proponents, including improving 
students’ performance and engagement in high school as well as their preparation for post- 
secondary education and work. 



II. Previous Research om Career Academies 

MDRC’s Career Academies Evaluation is built on a foundation laid by several earlier 
studies of Academies. Some of these have documented the feasibility and institutional growth of 
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the Career Academy approach in a range of local settings. 5 Other studies have included assess- 
ments of the Academies’ effects on student outcomes, such as graduation from high school, en- 
rollment in post-secondary education, and labor market participation. 6 

A number of studies have focused on the California Partnership Academies. 7 Several of 
these studies compared the performance of Academy students with that of other students in the 
same high schools who had similar demographic characteristics and prior records of low grades, 
high absenteeism, and disciplinary problems. The results indicated that the Academy students 
earned more credits and had significantly better attendance, grades, and graduation rates than 
students in the comparison groups. Other Partnership Academy studies have relied on school re- 
cords or survey data that compare Academy students with the general high school population. 
These studies also found that Academy students and graduates outperformed their non-Academy 
peers. 8 These results have been highlighted as particularly impressive, given that the state-funded 
Partnership Academies are required to recruit a majority of students who have been identified as 
economically or educationally disadvantaged. 

Outside California, several other studies using similar methodologies also reported posi- 
tive results. Evaluations of Academies in Philadelphia found that Academy students had higher 
attendance and graduation rates than the citywide average. 9 An evaluation of Academies affili- 
ated with the Junior Reserve Officers’ Training Corps (JROTC) found positive effects on atten- 
dance, credits earned, grades, and dropout prevention. 10 

Despite the broad array of research on Career Academies, a number of questions remain 
unanswered. Most important, several of the prior studies recognized that the positive effects they 
found may actually under- or overestimate the true effects of the Academies on student out- 
comes." On the one hand, it may have been that these effects were the result of the extra motiva- 
tion of students who were attracted to the Academies rather than to the unique experiences of- 
fered by the programs. In other words, the Academies may attract students (even students whose 
background characteristics may indicate a risk of dropping out) who are motivated enough to do 
well under a wide range of circumstances. On the other hand, because many Academies explic- 
itly attempt to serve at-risk students, the true effects of the programs may be understated in cases 
where such students are being compared with the general high school population, which includes 
many highly engaged and high-performing students. Many who conducted prior evaluations of 
Career Academies have emphasized that a random assignment research design would be neces- 
sary to eliminate these types of concern. In recognition of this, the California state legislature 
passed legislation in 1993 authorizing an evaluation of the Academies using random assignment 
of students under appropriate circumstances. The governor subsequently approved the present 
study as that evaluation. 



5 Snyder and McMullan, 1987; Stem, Raby, and Dayton, 1992; Academy for Educational Development, 1989, 
1990; Pauly, Kopp, and Haimson, 1995; Stem, Finkelstein, Stone, Latting, and Domsife, 1994. 

6 For the most comprehensive summary of this research, see Stem, Raby, and Dayton, 1992. 

7 Stem, Dayton, Paik, and Weisberg, 1989; Stem, Raby, and Dayton, 1992. 

8 Maxwell and Rubin, 1997, 1999; Dayton, 1997;Reller, 1987. 

9 Snyder and McMullen, 1987; Academy for Educational Development, 1989; Linnehan, 1996. 

'“Hansner, Elliott, and Gilroy, 1999; Hanser and Stasz, 1999; Stasz, 1999. 

"Stem, Raby, and Dayton, 1992. 
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A second important set of questions that has not received much attention concerns the 
variation in Academy effectiveness across a wide range of contexts and networks. On a related 
issue, little is known about the relative effectiveness of Academies for several key subgroups of 
students, such as those at high risk of dropping out of high school versus those highly likely to be 
college bound. MDRC’s Career Academies Evaluation is positioned to answer these questions 
and fill other gaps in the research on Academies. 



III. A ComceptniaB Framework of the Career Academy Approach 
and Its Potential Impact on Stmdeimfr Outcomes 

The key goal of this evaluation has been to assess the extent to which Career Academies 
keep students engaged in school, help them progress toward graduation, and prepare them for 
post-secondary education and work. In an effort to shed light on how and why the Academies do 
or do not affect changes in these outcomes, the evaluation has attempted to articulate a concep- 
tual framework, or theory of change, for the Career Academy approach. This framework identi- 
fies many of the key goals that have been proposed for Academies and attempts to make explicit 
some of the pathways through which the core elements of the approach are likely to improve out- 
comes for students. It has also been used to guide the design and implementation of several key 
features of the evaluation, including site selection, data collection, and analyses strategies. 12 

Figure 1.1 is a simplified conceptual model listing the basic Career Academy elements on 
the left and showing the hypothesized pathways through which these elements are likely to affect 
student outcomes during high school and beyond. The conceptual model covers four sets of con- 
structs delineated by the four columns in Figure 1.1 : 

• Career Academy organizational elements that distinguish the Academy ap- 
proach from the regular high school environments in which it is implemented; 

® supports and learning opportunities that are intended to evolve from the or- 
ganizational elements; 

• high school outcomes that the Academies aim to improve by enhancing the 
supports and learning opportunities in the previous column; and 

® post-secondary outcomes that reflect some of the long-term goals of the 
Academy approach. 



l2 While the concept of grounding program evaluations in theories of change is not new (see Weiss, 1995, for a 
detailed discussion of theory-based evaluation strategies), this evaluation marks the first attempt to make the theory 
behind Career Academies more explicit and then to collect and analyze data to examine empirically the hypotheses 
embedded in the theory. As such, the conceptual framework articulated in this report, and in the previous reports 
from the study, does not necessarily reflect a previously agreed-upon set of program characteristics and underlying 
principles on which Career Academies have been planned, implemented, and sustained. As noted earlier, only re- 
cently has a commonly agreed-upon definition of a Career Academy been articulated and disseminated. Also, the 
history and diversity of Career Academies highlights the fact that the goals of the approach are broad and evolving. 
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Linkages among these sections of the framework highlight the pathways through which the 
Academies are hypothesized to affect students’ experiences and behaviors. The conceptual 
framework is described briefly below. 

A. Career Academy Organizational Elements 

School-Witlhin-a-School Organization. In this organizational arrangement, clusters of 
students share several classes each day and often have the same small group of teachers from 
year to year. The student clusters vary in size but usually range from 30 to 60 students per grade 
in grades 9 through 12 or in grades 10 through 12. The number of classes students take within an 
Academy, and thus the number of teachers they share, also varies from Academy to Academy 
and from year to year, but usually students take from two to seven Academy classes. Teachers, 
who come from a variety of academic and vocational disciplines, are scheduled to have mostly 
Academy students in their classes. These teachers make a commitment to meeting with each 
other on a regular basis, and they share in decision-making related to administrative policies, cur- 
riculum content, and instruction. One teacher usually assumes lead responsibility for administra- 
tive tasks and serves as a liaison with the school principal and other administrators, school dis- 
trict officials, and employer partners. Students also take some regular classes along with the other 
students in the high school, and all courses in the Academy are counted as credits toward a high 
school diploma. Academy classes are often scheduled in blocks of three or four during the 
morning, leaving the remainder of the day for regular courses. This block scheduling allows for 
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special activities during this time: field trips, for instance, or team teaching, or hosting speakers 
from the business community. Teachers also attempt to involve parents in the Academy program, 
and schools often require parents to attend meetings with their children. 

Academic and Vocational Curricula Based on Career Theme. The Career Academies’ 
curricula usually consist of three or more academic courses per year and at least one vocational 
or occupation-related course per year that focuses on the selected career theme. These classes 
enable students to meet high school graduation and college entrance requirements and, at the 
same time, provide them with marketable skills. Students take their remaining course require- 
ments and electives (usually 20 to 50 percent of the credits needed to graduate) outside the Ca- 
reer Academy in the regular high school. To link the academic and occupational classes, Acad- 
emy teachers work together to coordinate course content and instructional strategies. They also 
focus on providing instruction in employability skills, both in the occupational theme courses 
and in one or more academic courses. Occupational classes are structured around whole indus- 
tries: Health Academies, for example, attempt to expose students to diverse medical occupations 
in the areas of direct care, technology, and administration. The Academy’s career theme is typi- 
cally chosen on the basis of local employment needs and demand for expertise in the national 
marketplace. 

Employer Partnerships. Career Academies strive to build formal relationships with a 
group of employers in their community. In general, the employer partnerships can be defined 
as ongoing coordinated efforts to engage local employers in supporting the Academy’s pro- 
grams and sponsoring a range of work- and career-related activities for students. The partner- 
ship typically includes employer representatives, teachers, school administrators, parents, and 
students. Many Academies create formal advisory boards that provide guidance on curricular 
and extracurricular activity development and may even assist with the management and ad- 
ministration of the program. Employer partners typically support the Academies by providing 
additional material resources or even making financial contributions. Most important, how- 
ever, is that employer partners contribute the time for their employees to appear as guest 
speakers in the school, supervise student internships, serve as mentors for individual students, 
and provide other kinds of support. 

Many Career Academies designate staff who serve as liaisons between the employers and 
the Academies and coordinate the various employer-sponsored activities. This role is crucial in 
creating and sustaining the various career development and awareness activities that are offered 
to students, both in school and outside school. The people in this role also take responsibility for 
developing work-based learning opportunities for students and monitoring student involvement 
in these activities. In some cases, this role is filled by Career Academy teachers who also have 
classroom responsibilities (although, usually, with a reduced course load). In other cases, the role 
is filled by non-teaching administrators whose primary responsibilities focus on one or more 
Academies. 

B. Su pports and Learning Opportunities 

The basic organizational elements of the Career Academy approach have particular ap- 
peal because they offer direct responses to several common structural problems that have been 
identified in high schools, particularly schools serving low-income communities and students 
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placed at risk of school failure. 13 These core elements should be viewed as institutional mecha- 
nisms that are mutually reinforcing and, together, are intended to facilitate enhancements of in- 
terpersonal supports and enriched teaching and learning opportunities. 

The second column of Figure 1.1 lists the types of supports and opportunities that are hy- 
pothesized to evolve from each of the three core organizational elements: (1) enhanced interper- 
sonal support through the intensive interaction and collaboration offered by the school-within-a- 
school; (2) a focused curriculum and enriched teaching and learning experiences through the in- 
tegration of academic and occupational content; and (3) exposure to career awareness and work- 
based education through the employer partnerships. These are discussed briefly below. 

Interpersonal Supports. Career Academies aim to function as “communities of support” 
for students and teachers. For students, such support includes the personalized attention they get 
from their teachers, their teachers’ expectations of them, their classmates’ level of engagement in 
school, and the opportunities they have to collaborate with their peers on school projects. Acad- 
emy teachers in this study indicated that they were supported by, among other things, opportuni- 
ties for professional collaboration and development, adequate resources, the capacity to influence 
instructional and administrative decisions, and opportunities to give personalized attention to 
students. Both this study and previous research have identified these dimensions of support as 
factors that can have important effects on both students’ motivation and engagement in school 
and teachers’ job satisfaction and sense of having an impact on students’ lives. 

Focused Curricula aod Enriched Learning Opportunities. Key goals of the Career 
Academy curricula are to ensure that students meet the core academic requirements they need to 
graduate and prepare for college and to focus students’ non- Academy course-taking on a coher- 
ent set of vocational or occupation-related classes. In addition, curricula attempt to provide stu- 
dents with applied learning opportunities, including developing problem-solving skills, using com- 
puters or manipulative materials, working on long-term projects, and connecting what they learn to 
other subjects or the world of work. These types of activities have been identified as strategies for 
breaking down the dichotomy between academic and vocational curricula, or between abstract and 
applied learning. Such a dichotomy is often seen as a structural feature of high schools that limits 
their capacity to help students make the transition from school to work or to post-secondary educa- 
tion. Previous research has suggested that the separation of curricular tracks has created a false and 
unnecessary dichotomy between academic rigor and real-world relevance. 14 

Career Awareness and Work-Based Learning Opportunities. In general, Career 
Academies aim to provide students with two types of work-related learning opportunities that 
are built on the employer partnerships. The first, referred to as career awareness and develop- 
ment activities, are intended to enhance students’ understanding of the world of work in gen- 
eral as well as their awareness of occupations within the program’s broad career theme. Some 
of these activities occur outside school. They include field trips designed to expose students to 
various work environments and to provide opportunities to observe a regular workday. An- 
other example is job-shadowing, which gives students the opportunity to accompany an adult 
on her or his job for a day or more. Some Career Academies develop mentoring programs to 
help students make connections with caring adults who can provide personal support and ca- 

13 See Kemple and Rock, 1996; Kemple, 1997a. 

14 Berryman, 1991; Raizen, 1989; Resnick, 1987; Dewey, 1916. 
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reer guidance. Other career awareness and development activities occur in school. These in- 
clude formal and structured attempts to infuse Career Academy classes with discussions and 
activities focusing on careers or students’ work-based learning experiences. They also include 
career counseling and the formal and informal discussions students have with their teachers 
and peers regarding preparation for work. 

The second type of work-related learning opportunities are typically referred to as work- 
based learning, which has been defined as “work experiences that are planned to contribute to 
the intellectual and career development of students.” 15 Work-based learning activities are proba- 
bly the most intensive and distinctive work-related aspect of the Career Academy approach. Stu- 
dents are typically placed in jobs that have been developed in collaboration with the employer 
partners and are connected to school. Students have the opportunity to learn both job-specific 
skills and more general work habits and behaviors. 

C. High School Outcomes 

The supports and opportunities listed in the second column of Figure 1.1 are also mutu- 
ally reinforcing and, together, are intended to affect students’ engagement, performance, and de- 
velopment during their high school years The third column of the figure summarizes several key 
student outcomes that Career Academies are intended to improve. These include preventing stu- 
dents from dropping out of high school, helping them meet graduation requirements, enhancing 
their achievement, helping them meet college entrance requirements, providing necessary steps 
to apply for and be accepted into college or a job, promoting constructive use of non-school 
hours, and reducing risk-taking behaviors. 

As noted earlier, this report focuses on the extent to which the Career Academies change 
these and other outcomes during students’ high school years. Analyses presented later in the re- 
port will explore how particular organizational features of the Academies — or the types of sup- 
ports or learning opportunities that develop from them — may or may not help account for the 
program impacts or lack of impacts. 

D. Post-Secondary Outcomes 

As shown in Figure 1.1, graduating from high school and acquiring various credentials 
should be viewed as transitional outcomes — as indications of students’ level of preparedness for 
future education and work after high school. Ultimately, as indicated in the fourth column of the 
figure, the Career Academies are intended to lead to higher levels of post-secondary education 
and to higher-skilled and higher-paying careers. 

Future reports from the Career Academies Evaluation will examine the impact Acade- 
mies may have on these types of outcomes and will explore the connections between high school 
experiences and the impacts that may accrue after high school. 



15 Office of Technology Assessment, 1995, p. 13. 
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IV. Kev Features of the Career Academies Evalmaitiom 

In 1993, MDRC began development work for a unique study of the Career Academy 
approach. Its primary purpose has been to provide reliable evidence about the efficacy of the 
theories and hypotheses embedded in the conceptual framework illustrated in Figure 1.1. The 
evaluation responds to the growing demand for rigorous evidence about the effectiveness of 
school-to-work and other high school reform initiatives. This section of the chapter describes 
the key features of the evaluation design that are central to understanding the findings pre- 
sented in this report. 

A. The Random Assignment Design 

The Career Academies Evaluation is a rarity in the field of education research in that it 
has been built on a random assignment research design and demonstrates the feasibility of im- 
plementing such a design within an ongoing high school program. In order to assess the differ- 
ence that Career Academies make in the lives of high school students, the research design in- 
volves a comparison between students who applied for and were randomly selected to enroll in 
a Career Academy and students who also applied but were not selected. During the evaluation 
period, a random selection process, or lottery, was used to make the final selection of students 
for the Academies. This was possible because each Academy had more qualified applicants 
than it was able to serve and the sites were willing to implement the protocols called for in the 
random selection process. 16 

This report focuses on a sample of 1,764 students from nine of the sites selected for the 
study. 17 For the purposes of this report, this group of students is referred to as the study sample. 
Of the students in the study sample, 959 (54 percent) were randomly selected to enroll in an 
Academy. For the purposes of this report, these students are referred to as the Academy group. 
The remaining 805 students (46 percent of the study sample) were not invited to participate in the 
Academies but could choose other options available in the high school or school district. These 
students constitute the study’s control group and are referred to in this report as the non-Academy 
group. In most cases, non-Academy group students enrolled in the general programs in the par- 
ticipating high schools, but in some cases they enrolled in citywide magnet programs or schools. 

Figure 1.2 illustrates the random assignment research design and shows the comparison 
being made to determine the impact Career Academies have on high school outcomes. The boxes 
on the lower right side of Figure 1.2 correspond to the first three columns of Figure 1.1. The first 
box represents the distinctive organizational features of the Academies, and the second box rep- 
resents the resulting supports and learning opportunities that derive from those features. The third 
box indicates the high school outcomes achieved by students randomly selected for the Academy 
group. Each of these boxes has a counterpart on the left side of Figure 1.2 for students randomly 
selected for the non-Academy group. The differences in outcomes between the two groups of 
students represent impacts of the Career Academies. 



l6 See Chapter 4 in Kemple and Rock, 1996, for a more detailed description of how the random assignment pro- 
cedure was implemented for this study. 

17 One of the 10 initial Career Academies was disbanded after two years in the study and was unable to provide 
sufficient follow-up data for its students in the study sample. 
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Figure 1.2 

Career Academies Evaluation 

Random Assignment Design 
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The random selection process ensured that the two groups of students were virtually the 
same on average in terms of their background characteristics, prior school experiences, and initial 
motivation and attitudes toward school. Any systematic differences in the outcomes that subse- 
quently emerged between the groups resulted from differences in their access and exposure to the 
Career Academies. 18 Differences in these school environments should have produced differences 
in the types of supports and learning opportunities experienced by students in the Academy and 
non- Academy groups. In fact, the previous reports and papers from the evaluation provide exten- 
sive evidence that Academy students experienced significant enhancements in the supports and 
learning opportunities illustrated in Figure 1.1, compared with their non- Academy counterparts. 
Academy students were also more likely to participate in a broad range of career awareness and 
work-based learning activities. Finally, Academy students were also somewhat more likely than 
their non- Academy counterparts to be exposed to various enriched learning activities in the class- 
room, such as applied learning and work-related problem-solving activities. The current report 
addresses the question of whether the Academies produced impacts on high school outcomes as 
represented by the three boxes at the bottom of Figure 1.2. 

B. Sites in the Career Academy Evaluation 

A second critical feature of the Career Academies Evaluation research design involves 
the selection of sites for participation in the study. MDRC was primarily interested in selecting 
sites that had already implemented versions of the organizational elements listed in the first col- 
umn of Figure 1.1. 19 This was important because a key goal of the evaluation was to include 
functioning Academies that encompassed the central elements of the approach, rather than pro- 
grams that were in the initial stages of implementation. 

In addition, MDRC sought high schools in which there was a clear contrast between the 
Career Academy and other programs available to potential Academy students. This was impor- 
tant because one of the primary concerns of the study was whether Career Academies improve 
students’ post-secondary education and employment outcomes above and beyond what would 
have occurred had they not had the opportunity to attend an Academy. Some high schools and 
school districts operate more than one Career Academy or other Academy-like programs. In such 
circumstances, many students in both groups would likely be involved in similar programs. This 
would reduce the contrast between their experiences and could mistakenly obscure the real ef- 
fects of the Academies and make it appear that the Academies were ineffective. 

Each participating site had established the basic Career Academy components described 
in this chapter: a school-within-a-school organization, academic and vocational curricula based 
on a career theme, and employer partnerships. This combination of features was not available 
elsewhere in the participating high schools. 20 Each Career Academy attempted to serve a wide 



l8 As discussed in Chapter 2, not all students randomly selected for the Academy group actually enrolled and 
remained in an Academy, and a small percentage of students selected for the non-Academy group did enroll. 

,9 The site selection process is described in greater detail in Kemple and Rock, 1996, which also includes a de- 
tailed description of the Career Academies in the sites. 

“Although some participating high schools do operate other programs that they classify as Career Academies, 
information collected for this study indicated that most such programs do not include the basic components of the 
Academy approach described earlier. As a result, the participating Career Academy programs represent a clear con- 
trast with other programs in the high schools. 
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range of students, including those with a history of poor school engagement and performance as 
well as those who were engaged in school at the time they applied for the programs. 

The participating Academies offer a range of occupational themes: three are in the busi- 
ness and finance fields; three focus on high-technology areas such as electronics and aerospace 
technology; and there is one each in the fields of health occupations, public service, travel and 
tourism, and video technology. The participating programs were drawn from most of the major 
established networks of Career Academies across the country, with four from the California 
Partnership Academy network, two from the National Academy Foundation network, one from 
the Florida network of Academies for Career Development and Applied Technology, and one 
from the network of Academy programs created by the District of Columbia Public Schools. 
Two of the participating Academies were developed independently through local high school or 
district initiatives. 

As of the 1994-95 school year (when the last sites joined the study), the participating Ca- 
reer Academies had been in operation for as few as two years and as many as 1 1 years. Nine of 
the 10 Career Academies remained in operation throughout the evaluation period and were able 
to meet the data and other research-related needs of the evaluation. One site was disbanded after 
the 1995-96 school year and was unable to meet the data needs of the evaluation. 

In summary, the sites participating in the Career Academies Evaluation provide a solid 
foundation on which to build a credible assessment of the implementation and impact of the Ca- 
reer Academy approach. Three important cautions should be kept in mind, however, in inter- 
preting the findings from this study and, in particular, the findings presented in this report. 

First, because the participating sites were chosen strategically, rather than randomly, the 
findings from this study cannot necessarily be generalized to all schools and school districts. 
These are school districts and high schools that were willing and able to commit the financial and 
personnel resources needed to implement and sustain a Career Academy. At the same time, how- 
ever, these sites, as a group, share the characteristics of typical urban and small-city school dis- 
tricts, and, individually, they reflect much of the diversity of such districts. This provides some 
basis for extending the findings and lessons from this study beyond the participating schools. 

Second, like their host high schools and school districts, the participating Career Acade- 
mies are dynamic and evolving. Over the course of the evaluation, they have had to confront staff 
turnover, increases or decreases in funding, changes in local or state education policy, shifting 
levels of support from building or district staff, and changes in the amount and types of support 
they receive from employer partners. Because this is a longitudinal study, it has been able to pro- 
vide a realistic picture of how ongoing programs evolve and change in the context of dynamic 
high schools. In general, most of the programs modified various components of the Career Acad- 
emy approach in response to changing conditions in their host high schools or school districts, 
and many of them evolved toward more complete versions of the model. It should be noted, 
however, that some of the programs were weakened by staff turnover, funding reductions, or de- 
creased support from school staff or employer partners. As noted earlier, one Academy was dis- 
banded at the end of the 1995-96 school year. Although this site provides some useful lessons 
about institutional stresses that are likely to affect the sustainability of Career Academies, its dis- 
solution and lack of comparable data prevent it from providing information to assess its impact 
on student outcomes. 
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Third, the previous reports from MDRC’s Career Academies Evaluation focused pri- 
marily on findings that were aggregated across all participating sites. Although such findings 
shed light on the implementation and impact of the Career Academy approach more generally, 
they mask the high degree of variation among the sites and the ways this variation may be as- 
sociated with differences in program effectiveness. For example, some sites were able to de- 
velop a particularly cohesive school-within-a-school, while others made strong investments in 
their employer partnerships. More important, the Academies in some sites represented an es- 
pecially dramatic contrast with the regular school environment in terms of the degree of inter- 
personal and instructional support they offered students. A central focus of this report, there- 
fore, is on determining whether some versions or contexts for the Academy approach are more 
effective than others. 

C. Data Used in This Report 

The conceptual framework described above has helped guide data-collection activities for 
the evaluation. For example, MDRC researchers conducted a variety of field research activities to 
document and describe the organizational features illustrated in the first column of Figure 1.1. 
Several surveys were developed and administered to students and teachers to gain systematic in- 
formation about various supports and learning opportunities that might be captured by the second 
column. MDRC has also collected data from school records and students’ transcripts to obtain 
information about their progress and performance in high school. As part of the evaluation, 
MDRC administered standardized math and reading tests to a subsample of students, and it con- 
ducted a survey at the end of students’ 1 2 th -grade year to learn about their use of non-school 
hours and preparation and plans for the future. Finally, as the evaluation moves forward, MDRC 
will continue to follow students beyond their high school years to collect information about their 
progress in post-secondary education and the labor market. 

The primary data for this report were obtained from three sources: school transcript rec- 
ords, a survey that students in the study sample completed at the end of their ^^-grade year, and 
a standardized math computation and reading comprehension test administered to a subsample of 
the students at the end of their 12 th -grade year. These are described briefly below. 21 

School Transcript Records. A complete set of school transcript records is available for 
1 ,454 students in the study sample. 22 This is referred to in the report as the Student School Rec- 
ords Database. The Student School Records Database includes information about students’ daily 
attendance rates, credits earned toward graduation, and course-taking patterns. Of the students in 
the Student School Records Database sample, 1,293 remained enrolled in high school through 



21 See Appendix A for a more detailed discussion of the response rates and analytical issues associated with the 
data-collection efforts for the evaluation. 

22 MDRC attempted to collect school transcript records for all students in the study sample even if they had 
transferred to other high schools within the districts in which the participating Career Academies were located. 
MDRC was not able to obtain school transcript records for students who transferred to high schools outside these 
districts. Analyses of differences in data availability among students in the study sample indicated that there were no 
systematic differences in school records availability between Academy and non-Academy group students. MDRC 
obtained school records data for 82 percent of students in the Academy group and for 84 percent of students in the 
non-Academy group. Among students in the Student School Records sample, there were no systematic differences 
in the background characteristics of Academy and non-Academy group students. These analyses provide greater 
confidence that the Student School Records Database will yield valid estimates of Career Academy impacts. 
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the end of their ^"’-grade year. The remaining 161 students (12 percent) were confirmed to have 
dropped out of high school before the end of their 12 th -grade year. A complete set of school tran- 
script records was obtained for these students up to the point at which they dropped out. 

12th Grade Survey. The 12th Grade Survey was completed by 1,510 students in the 
study sample at the end of their 1 2 lh -grade year. 23 This is referred to as the 12th Grade Survey 
Database. The 12th Grade Survey asked students a wide range of questions about their school 
experiences, employment and work-related experiences, extracurricular activities, preparation for 
college and post-secondary jobs, and plans for the future. 

Achievement Test Scores. Math computation and reading comprehension achievement 
tests were administered to 490 students in the study sample. 24 This is referred to as the 1 2 th Grade 
Achievement Test Database. This test was initially designed by the Educational Testing Service 
(ETS) for the National Educational Longitudinal Surveys of 1988 (NELS: 88) follow-up and was 
administered to a nationally representative sample of students in their 12 th -grade year. ETS per- 
mitted MDRC to administer the test to a subsample of students in the Career Academies Evalua- 
tion sample. 25 The Achievement Test Database provides national percentile scores both in read- 
ing comprehension and in math computation and problem-solving. It also provides criterion- 
referenced scores that indicate whether students demonstrated proficiency at particular skill lev- 
els of math and reading. 



V. Overview of This Report 

A central theme that has emerged from this evaluation is that an accurate and useful 
assessment of the effectiveness of the Career Academy approach must recognize that the pro- 
gram is not a “one-size-fits-all” initiative and that it produces different impacts for different 

23 MDRC attempted to survey all students in the study sample even if they had transferred to high schools out- 
side the participating districts or had dropped out of high school altogether. Analyses of differences in data avail- 
ability among students in the study sample indicated that there were no systematic differences in 1 2th Grade Survey 
response rates between Academy and non-Academy group students. MDRC obtained completed surveys from 86 
percent of students in the Academy group and from 85 percent of students in the non-Academy group. Among stu- 
dents in the 12th Grade Survey sample, there were no systematic differences in the background characteristics of 
Academy and non-Academy group students. These analyses provide greater confidence that the 12th Grade Survey 
Database will yield valid estimates of Career Academy impacts. 

24 MDRC attempted to administer the achievement test to the 69 1 students in the study sample who were sched- 
uled to be in 12 lh grade at the end of the 1997-98 school year. The 490 students who completed the achievement test 
represents 71 percent of those attempted. Analyses of differences in data availability among students in the study 
sample indicated that there were no systematic differences in achievement test completion rates between Academy 
and non-Academy group students who were attempted. MDRC obtained completed achievement tests from 72 per- 
cent of the Academy group students attempted and from 70 percent of the non-Academy group students attempted. 
Among students in the Achievement Test sample, there were no systematic differences in the background charac- 
teristics of Academy and non-Academy group students. These analyses provide greater confidence that the 
Achievement Test Database will yield valid estimates of Career Academy impacts. 

25 The achievement test was administered to students on a Saturday morning near the end of their 1 2 th - grade 
year. They were offered a stipend of $50 if they completed the test. Some concerns have been raised about whether 
this test instrument and the conditions under which it was administered provide an adequate indication of student 
achievement in math computation and reading comprehension. It should be noted, however, that the same test was 
administered under similar conditions as part of the U.S. Department of Education’s National Educational Longitu- 
dinal Surveys of 1988 (NELS:88). 
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types of students. In order to highlight the importance of this theme, Chapter 2 describes the 
students who are in the study sample and identifies subgroups of students who are most likely 
to derive various benefits from the programs. All the key analyses and findings presented in 
this report are broken down by these subgroups. Chapter 2 also presents findings from an 
analysis of the patterns by which students in the study’s Academy group enrolled and re- 
mained in the Career Academy programs. 

Chapter 3 assesses the impact Career Academies have on students’ high school engage- 
ment and performance and on their preparation for post-secondary education and employment. It 
focuses first on the impacts Career Academies produce for students who were at high risk of 
dropping out of high school. It then presents the results for students who entered the study highly 
engaged in school and were at very low risk of dropping out. Finally, it presents the impact 
findings for students who fell into a middle range of characteristics associated with a risk of 
school failure. 

As noted earlier, this report also attempts to push much further in offering lessons about 
the efficacy of the Career Academy approach and other school reform initiatives that are embed- 
ded in it. The contrast among the sites and the differences in effectiveness for key subgroups of 
students provide a rich context for making judgments about what types of school reform initia- 
tives are likely to be effective and for whom. Chapter 4 presents findings from an analysis that 
begins to highlight some of the key mechanisms by which the Academies may produce the im- 
pacts described in Chapter 3. This chapter identifies a group of sites, within the evaluation sam- 
ple, in which the Career Academies represented a particularly dramatic contrast with their non- 
Academy school environments in terms of some of the key supports and learning opportunities 
described earlier. It also identifies a second group of sites that had very little contrast between the 
Academy and non-Academy school environments in these areas. Chapter 5 examines variation in 
impacts across these two groups of sites. 
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Chapter 2 



Career Academy Students and 
Their Patterns of Enrollment In the Academy Programs 



This chapter describes the background characteristics and prior school experiences of the 
students in the research sample for this report. It also summarizes findings on the patterns by 
which those selected for the Career Academies actually enrolled and remained in the programs. 
The chapter makes two key points. 

First, the chapter highlights the fact that the students in the study sample come from di- 
verse backgrounds and prior school experiences. This suggests that analyses that do not account 
for this diversity are likely to mask variation in the difference that Career Academies may make 
for some students and not for others. Section II of this chapter, therefore, identifies subgroups of 
students defined by background characteristics and prior school experiences associated with dif- 
ferent patterns of school success or failure. The impact findings presented in Chapter 3 show that 
the Career Academies produced quite difference patterns of impacts for these three subgroups of 
students. 

Second, the analyses presented in this chapter show that 88 percent of the students ran- 
domly selected for the study’s Academy group (and invited to enroll in a Career Academy pro- 
gram) actually enrolled. By the end of 12 th grade, 59 percent of the students initially selected for 
the programs were still enrolled in them. These enrollment and attrition patterns have implica- 
tions for Career Academy policies and implementation practices. They also provide an important 
context for interpreting the impact findings presented later in the report. 



I. Students in the Study SampBe for This Report 

This section of the chapter describes the background characteristics of the 1,764 students 
who constitute the study sample for this report. The description highlights the fact that no sys- 
tematic differences were found in the background characteristics of the Academy group and the 
non-Academy group. This is the central purpose of the random selection process used in creating 
these two groups, and it extends to measured as well as unmeasured characteristics. 

A. Background Characteristics of Students im the Study Sample 

The Career Academies Evaluation has included efforts to collect information about 1,764 
students who applied for one of nine Career Academies across the country between 1993 and 
1996.' In this report these students are referred to as the study sample. Table 2.1 lists a variety of 



'See Kemple and Rock, 1996, for a more detailed discussion of the procedures used to select students for the 
Career Academies Evaluation study sample. The initial sample for the Career Academies Evaluation consisted of 
1,953 students from 10 sites. As noted earlier, one of the initial sites was disbanded and was not able to provide 
follow-up information needed for the analyses in this report. Thus, the 126 students in the initial study sample from 
that site are not included in the analyses. Also, MDRC found that information could not be obtained for 59 of the 
initial group of students because they should not have been included in the study sample. Four other students were 
found to be deceased. 
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Table 2.1 

Career Academies Evaluation 



Background Characteristics off Study Sample, 
by Research Status 



Characteristic 


Full Sample 
(%) 


Academy 

Group 

(%) 


Non-Academy 

Group 

(%) 


Bemoeraohic and family characteristics 
Gender 


Male 


43.8 


44.6 


42.9 


Female 


56.2 


55.4 


57.1 


Age of student at time of application 


1 3 or younger 


8.6 


7.3 


10.1 


14 


35.6 


35.7 


35.5 


15 


46.1 


46.8 


45.2 


1 6 or older 


9.7 


10.1 


9.2 


Race/ethnicity 


Black 


30.2 


30.9 


29.4 


White 


6.4 


6.0 


6.9 


Hispanic 


56.2 


55.5 


57.0 


Asian or Native American 


7.2 


7.5 


6.7 


Student speaks limited English 3 


7.6 


7.0 


8.3 


Student lives with 


Mother and father 


61.7 


61.1 


62.5 


Mother only 


28.6 


29.0 


28.1 


Father only 


4.6 


5.0 


4.1 


Other family/nonrelative 


5.1 


4.9 


5.4 


Student lives in single-parent household 


38.3 


38.9 


37.5 


Father's education level 


Did not finish high school 


39.8 


38.9 


40.9 


High school graduate 


32.4 


32.2 


32.6 


Completed some post-secondary 


27.8 


28.9 


26.6 


Mother's education level 


Did not finish high school 


36.1 


35.2 


37.1 


High school graduate 


34.8 


34.5 


35.3 


Completed some post-secondary 


29.1 


30.3 


27.6 


Neither parent has high school diploma 


28.6 


29.0 


28.2 


Parental Work 


Both parents work 


47.3 


46.5 


48.3 


Father works 


23.8 


23.5 


24.1 


Mother works 


17.8 


19.2 


16.2 


Neither parent works 


11.1 


10.8 


11.4 



(continued) 
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Table 2.1 (continued) 



Characteristic 


Full Sample 
(%) 


Academy 

Group 

(%) 


Non-Academy 

Group 

(%) 


Family receives welfare or Food Stamps 


24.2 


23.6 


25.0 


Family mobility in past two years 


Have not moved 


59.4 


58.8 


60.2 


Moved 1 or 2 times 


33.6 


34.8 


32.2 


Moved 3 or more times 


7.0 


6.5 


7.6 


Student is home alone more than 3 hours per day 


13.5 


13.5 


13.6 


Educational characteristics 

8 th -grade math test score 0 


75 l percentile or higher 


8.5 


8.8 


8.1 


50 th to 74 th percentile 


20.4 


21.0 


19.7 


25 th to 49 th percentile 


32.2 


29.9 


35.0 


24 th percentile or lower 


38.9 


40.3 


37.2 


8 th -grade reading test score c 


75 th percentile or higher 


9.8 


10.4 


9.0 


50 th to 74 th percentile 


19.4 


20.8 


17.7 


25 th to 49 th percentile 


36.3 


33.7 


39.4 


24 th percentile or lower 


34.6 


35.1 


33.9 


Student does not feel safe at school 


23.2 


22.7 


23.9 


Frequency of cutting classes 


Never 


78.9 


79.5 


78.3 


At least 1 time a week 


19.7 


19.4 


20.1 


Daily 


1.4 


1.2 


1.6 


Sent to office for misbehavior 


Never 


81.3 


81.0 


81.6 


1-2 times 


15.7 


16.2 


15.2 


3-10 times 


3.0 


2.8 


3.2 


Hours per week spent on homework 


1 hour or less 


28.8 


27.9 


30.0 


2-3 hours 


38.2 


39.3 


36.9 


4-6 hours 


17.4 


18.5 


16.0 


7 hours or more 


15.6 


14.3 


17.2 


Hours per day spent watching TV 


Less than an hour 


12.3 


11.7 ■ 


13.0 


1-2 hours 


27.1 


27.4 


26.7 


2-3 hours 


26.8 


24.9 


29.1 


Over 3 hours 


33.8 


36.0 


31.3 


Student has worked for pay 


36.3 


35.8 


36.9 



(continued) 
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Table 2.1 (continued) 







Academy 


. Non-Academy 




Full Sample 


Group 


Group 


Characteristic 


(%) 


(%) 


(%) 



Characteristics associated with dropping out of school 



Attendance rate, year prior to random assignment 



96-100% 


54.2 


53.1 


55.4 


91-95% 


24.1 


23.3 


24.9 


86-90% 


11.0 


12.2 


9.5 


85% or lower 


10.8 


11.4 


10.2 


Credits earned in 9 th grade d 


5 or more credits 


80.9 


80.4 


81.4 


3-4 credits 


13.7 


14.3 


12.9 


2 or fewer credits 


5.5 


5.3 


5.7 


Grade point average in year of random assignment 


3.1 or higher 


36.2 


34.4 


38.3 


2. 1-3.0 


38.1 


39.5 


36.5 


2.0 or lower 


25.7 


26.1 


25.2 


Student is overage for grade level 1 


21.1 


21.5 


20.7 


School mobility 8 


0 or 1 different school 


72.7 


72.9 


72.4 


2 or more different schools 


27.4 


27.1 


27.6 


Student has sibling who dropped out of high school 


20.2 


19.8 


20.6 


Sample size 


1,764 


959 


805 



SOURCES: MDRC calculations from the Career Academies Evaluation Student Baseline Questionnaire Database and 
Student School Records Database. 



NOTES: All characteristics were measured at the time students applied to the Career Academy program and prior to being 
randomly selected to the Academy and non-Academy groups. 

Invalid or missing values are not included in individual variable distribution. Rounding may cause slight discrepancies 
in calculating of sums and differences. 

A chi-square test was applied to differences in the distribution of characteristics across the Academy and non-Academy 
groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a These are students who responded that they spoke English "not well'' or "not at all." 

b Several different standardized, nationally normed math tests were administered to students, depending on the district 
where their school was located and the year they entered the study. National percentile scores were used because they were 
the only standardized scores available across tests. 

c Several different standardized, nationally normed reading tests were administered to students, depending on the district 
where their school was located and the year they entered the study. National percentile scores were used because they were 
the only standardized scores available across tests. 

d This was applicable only to students who applied to the Career Academy at the end of their 9 th -grade year. 

c Grade point averages were converted to a standard 4.0 scale from 100-point or 5-point scales for some sites. 

f A student is defined as overage for grade at the time of random assignment if she or he turns 1 5 before the start of the 
9 th grade, or 16 before the start of the 10 th grade. This indicates that the student was likely to have been held back in a 
previous grade. 

8 School mobility is defined as the number of schools attended since the 1 st grade beyond the number expected to result 
from promotions in grade level or graduations. 
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background characteristics and measures of prior school experiences for students in the study 
sample. The first column in the table lists the percentages of students in the full sample who had 
each of the characteristics listed. 

The first section of Table 2.1 indicates that students in the study sample come from a 
wide range of family backgrounds. The vast majority of students identified themselves as either 
Hispanic (56 percent) or black (30 percent). The relatively large proportion of Hispanic students 
reflects the fact that several of the sites are located in predominantly Chicano and Latino com- 
munities in California, south Texas, and south Florida. The black students are concentrated in 
several large eastern cities. 

The data collected for the evaluation does not include a direct measure of family income, 
but there are several indicators that the study sample includes a moderate proportion of students 
from low-income families. For example, Table 2.1 shows that 38 percent of the students lived in 
single-parent households at the time they applied for an Academy. Further, 1 1 percent of the stu- 
dents indicated that neither of their parents was working, and 24 percent indicated that their 
families were receiving welfare or Food Stamps. Twenty-nine percent reported that neither par- 
ent had received a high school diploma or a GED. Forty-one percent indicated that their families 
had moved at least once during the two years prior to applying for the Academy. The table also 
indicates that many students may come from middle-income families. Forty-seven percent of the 
students reported that both parents were working, and many students indicated that at least one of 
their parents completed some post-secondary education. 

The second section of Table 2.1 lists a variety of indicators of students’ prior school en- 
gagement and performance. The vast majority of students indicated that they did not have disci- 
pline problems in school and had high aspirations for their education. For example, approxi- 
mately 80 percent reported that they had never skipped class during the second semester of 8 th or 
9 th grade, and about the same percentage reported that they had never been sent to the school of- 
fice for misbehavior. Also, although not shown in the table, virtually all the students reported 
that they expected to graduate from high school, and nearly two-thirds indicated that they ex- 
pected to graduate from college. 

At the same time, many students appeared to be struggling somewhat in school. Less than 
10 percent of the students had S^-grade math or reading test scores in the 75 th percentile or 
higher, while 35 to 40 percent had test scores below the 25 th percentile. Twenty percent of the 
students reported that they had cut class at least once per week during the second semester of 8 th 
or 9 th grade, and nearly 20 percent indicated that they had been sent to the school office for mis- 
behavior. The table indicates an interesting comparison between the time students reported 
spending on homework and the time they spent watching television. One-third reported that they 
spent four or more hours per week doing homework (about one hour per weekday), and 61 per- 
cent that they spent two or more hours per day watching television. 

The third section of Table 2.1 lists several demographic and school-related characteristics 
that have been found, both in prior research and in analyses conducted for this evaluation, to 
have particularly strong associations with later school engagement and performance. As dis- 
cussed later in this chapter, these characteristics were used to create subgroups of students for the 
impact analyses. The respective subgroups include students with markedly different prospects for 
school success. 
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Many of the students appeared to be highly engaged in school. Over half the students in 
the study sample (54 percent) had attendance rates over 95 percent in the year they entered the 
study, and another 24 percent had attendance rates between 91 and 95 percent. Eighty-one per- 
cent of the students who applied to the Academies as 9 th -graders had earned at least five credits in 
that year and would be considered to be on track toward graduation. 2 Over one-third of the stu- 
dents had a 3.1 grade point average or higher (approximately equivalent to a B average or higher) 
during the year they applied for an Academy. 

A significant percentage of students in the study sample appeared to be disengaged from 
school. Eleven percent might be characterized as chronic absentees with attendance rates of less 
that 86 percent in the year they applied for the Academy programs. Nearly 20 percent of the stu- 
dents had already fallen behind in course credits, earning less than five in the 9 th grade; 6 percent 
had fallen well behind, earning two or fewer course credits in the 9 th grade. Approximately 21 
percent of the students were overage for their grade level, indicating that they had been retained 
in a previous grade. About 27 percent of the students reported that they had transferred schools 
two or more times beyond the regular school transitions (such as from elementary to middle 
school or middle school to high school), indicating that their attachment to school may have been 
disrupted by family mobility or other reasons. Finally, about 20 percent of the students indicated 
that they had a sibling who had dropped out of high school. 

B. Characteristics of Sfademts in the Academy and Non-Academy Groups 

As noted in Chapter 1, each of the students in the study sample applied for and was de- 
termined to be eligible for enrollment in one of the participating Career Academies. Because the 
programs had attracted more eligible students than they were able to serve, a lottery was used to 
select students for enrollment in the Academies. Of the students in the study sample, 959 (54 
percent) were randomly selected to enroll in an Academy, and 805 (46 percent) were not selected 
for enrollment in an Academy but were eligible to enroll in other programs or classes in the host 
schools or school district. 

The second and third columns of Table 2.1 provide a comparison between characteristics 
of students selected for the study’s Academy group and those of students in the non-Academy 
group. The table indicates that there were no statistically significant differences between the 
background characteristics and prior school experiences of students in the two groups. This is a 
result of the random assignment design and can also be extended to characteristics that are not 
directly measured by the data collected for this evaluation. These include such constructs as ini- 
tial motivation, attitudes toward school, and other baseline attributes that may be associated with 
school engagement and performance. In other words, the random assignment process created two 
groups for which there were no systematic differences initially, in both measured and unmeas- 
ured characteristics. As a result, one can be confident that any systematic differences that 
emerged after random selection can be attributed to the fact that the Academy group was selected 
to enroll in the programs and the non-Academy group was not. This chapter also includes a dis- 
cussion of the rates at which students actually enrolled in the Academies and examines the extent 



2 It should be noted here that approximately 22 percent of the students were applying for the Academies as 8 lh - 
graders by virtue of the fact that the Academies began in 9 lh grade. These students were not included in calculations 
of credits earned in 9 lh grade. 
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to which students remained in the programs and, thus, were likely to be exposed to the full range 
of Academy experiences. 



II. Subgroups of Students Defined by Characteristics Associated with 

Dropping Out of High School 

Previous research on Career Academies has not typically examined the relative effective- 
ness of the approach among the different types of students the programs serve. Although findings 
from previous research have been useful, they do not recognize the fact that Career Academies may 
change certain outcomes for some students but not necessarily for others. The random assignment 
research design used in this evaluation provides a unique opportunity to assess the potential for 
the Academies to make a difference for various groups of students. As noted above, engagement 
and performance of students in the study’s non-Academy group provide the best indication of 
how students in the Academy group were likely to behave if they did not have the opportunity to 
enroll in an Academy. 

In particular, many students in the study’s non-Academy group were unlikely to drop out 
of high school, making it unlikely that the Academies could reduce dropout rates or increase ba- 
sic school engagement much further for them. An important question about these students, how- 
ever, is whether the Academies expanded (or at least did not limit) their opportunities to pursue a 
more rigorous curriculum or increased their preparation for post-secondary education and em- 
ployment. By contrast, other students in the non-Academy group (for example, those who had 
failed several courses in 9 th grade) were at relatively high risk of school failure and could be seen 
as having the potential to benefit from involvement in the Academies in a variety of ways, in- 
cluding being prevented from dropping out. 

Given the dramatic differences in outcomes for various subgroups of students within 
the non-Academy group, the primary focus of this report is on the impact Career Academies 
have for students at greater or lesser risk of dropping out of high school or of doing poorly, if 
they remain in high school. To assess this variation in impacts, students in the study sample 
were divided into three subgroups based on selected background characteristics and prior 
school experiences that were associated with dropping out of high school. This section of the 
chapter provides an overview of the strategy used to identify these subgroups and highlights 
the basic distinctions among them. 

Student subgroups were defined using six characteristics found to be strong predictors of 
dropping out among students in the study’s non-Academy group. These characteristics were all 
measured at the time students applied for a Career Academy and before they were randomly se- 
lected for the Academy or non-Academy group. Each of these characteristics has also been identi- 
fied in prior research as being highly correlated with dropping out of high school. 3 They include: 

® average daily attendance in the year the student applied for an Academy; 

o grade point average for the year the student applied for an Academy; 



3 For a review of research literature on background characteristics, measures of prior school performance, and 
other factors associated with dropping out of school, see Natriello, 1987, and Roderick, 1993. 
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o the number of credits earned toward graduation in 9 th grade (for students apply- 
ing for an Academy at the end of 9 th grade); 

o whether the student was overage for grade (indicating the student had been re- 
tained in a prior grade); 

o whether the student had a sibling who dropped out of high school; and 

o whether the student had transferred schools two or more times beyond the typi- 
cal school transitions. 

In other random assignment studies, subgroups have been determined by identifying sample 
members using one or more characteristics associated with a difference in the way they are likely to 
be treated by the program under study or in the outcomes they are likely to achieve without having 
access to the program. 4 In education research, subgroups have been created by distinguishing 
between students who have, for example, two or more characteristics associated with school fail- 
ure and those who have only one such characteristic or none. 5 Such methods might be called 
“risk-factor accumulation” strategies because they involve simply adding up particular indicators 
and defining the subgroups based on the number of risk -related characteristics a given student 
has. 



Risk-factor accumulation strategies, however, have some limitations. In particular, they 
give equal weight to each of the selected risk-related background characteristics and prior school 
experiences. As a result, they do not account for the fact that some characteristics are more highly 
associated with school failure than others. In addition, this strategy does not account for the fact that 
some students also have a number of related characteristics that are associated with school success 
and that may offset the risk associated with other characteristics. For example, some students may 
have failed several courses in 9 th grade even though they attended regularly and did not have other 
background characteristics associated with dropping out. As a result, various combinations of char- 
acteristics, along with different degrees of importance attached to some characteristics, may indi- 
cate a different degree of risk. 

Given these limitations, a more systematic approach was used for this report, in order to 
identify groups of students who were clearly distinct in terms of their likelihood of dropping out 
in the absence of access to a Career Academy. 6 In particular, the background characteristics and 
prior school experiences listed above were used to predict the probability that students in the 
non-Academy group would drop out of high school. This provided an estimate of the unique 
contribution that each characteristic made to predicting that these students would drop out. For 
example, attendance rates and credits earned toward graduation were found to be better predic- 
tors of dropping out than students’ being overage for grade. Also, the prediction model provided 
the opportunity to give more weight to different specifications of a characteristic. For example, 
the lower a student’s prior attendance rate, the more likely that he or she would drop out. Thus, 
students with very low attendance rates might be considered at high risk of dropping out, even 



4 See, for example, Friedlander, 1988. 

5 See, for example, NCES, 1990 and NCES, 1992. 

6 Appendix B provides a more detailed and technical discussion of the subgroup identification strategy used for 
this report, including some potential limitations of this approach. 
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though they had not been held back in a previous grade and did not have a sibling who dropped 
out of school. 

Because of the random assignment research design, this approach, based on these char- 
acteristics, best predicts which students in the Academy group would have dropped out in ab- 
sence of access to an Academy. Thus, students in the Academy group were sorted using the same 
average background characteristics that were used to predict dropping out for the non- Academy 
group. Following are brief definitions of the three risk subgroups, which are described further 
below. 

• High-risk subgroup: students in the study sample (approximately 25 percent 
of both the Academy and the non-Academy groups) with the combination of 
characteristics associated with the highest likelihood of dropping out 

• Low-risk subgroup: students in the study sample (approximately 25 percent 
of both the Academy and the non-Academy groups) with the combination of 
characteristics associated with the lowest likelihood of dropping out 

• Medium-risk subgroup: the remaining students in the study sample (ap- 
proximately 50 percent of both the Academy and the non-Academy groups) 
with a mix of characteristics indicating that they were not particularly likely to 
drop out but were not necessarily highly engaged in school 

A. Characteristics of Students in the Risk Subgroup s 

Table 2.2 presents selected characteristics of students in the three risk subgroups, includ- 
ing the background characteristics and prior school experiences used to define the subgroups. 
Because each of the characteristics used to define the subgroups was measured before students 
were randomly assigned to the two main study groups, there are no systematic differences in ob- 
served background characteristics between Academy and non-Academy groups within each of 
the three risk subgroups. 7 

Students in the High-Risk Subgroup. The third section of Table 2.2 lists the six char- 
acteristics that were used to define the risk subgroups. It shows, for example, that 43 percent stu- 
dents in the high-risk subgroup had a sibling who dropped out of high school and that 43 percent 
were overage for their grade level (indicating they had been held back in a prior grade). About 
one-third of these students could be classified as chronic absentees (having an attendance rate of 
85 percent or lower in the year they applied to an Academy), and 62 percent had a grade point 
average of 2.0 or lower (out of a possible 4.0). Also, over half of the students in the high-risk 



7 The initial prediction of dropping out was based on analyses using the non-Academy group. An index of aver- 
age characteristics of likely dropouts from the non-Academy group was then applied to the Academy group. Given 
the statistical properties of the analyses used, random differences in characteristics between the Academy and the 
non-Academy groups are likely to yield somewhat more accurate predictions of likely dropouts for the non- 
Academy group. This means that, in the high-risk subgroups, the dropout rate for non-Academy students may be 
artificially higher than the dropout rate for Academy students. Extensive analyses were conducted to identify the 
potential magnitude of this distortion. These analyses indicate that whatever distortion exists is negligible and did 
not change the pattern of impacts. This issue is discussed in greater detail in Appendix B. 
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Table 2.2 



Career Academies Evaluation 

Background Characteristics of Study Sample, 
by Subgroups Defined by Risk of Dropping Out of School 





High-Risk 


Medium-Risk 


Low-Risk 




Subgroup 


Subgroup 


Subgroup 


Characteristic 


(%) 


(%) 


(%) 


Demographic and family characteristics 








Gender 








Male 


43.0 


45.5 


41.3 


Female 


57.0 


54.6 


58.7 


Age of student at time of application 








1 3 or younger 


2.1 


10.6 


1 1.7 


*** 


14 


22.2 


39.2 


43.6 




15 


54.4 


42.8 


43.3 




1 6 or older 


21.3 


7.4 


1.4 




Race/ethnicity 








Black 


37.6 


29.9 


22.7 


* ** 


White 


4.7 


7.2 


6.8 




Hispanic 


51.8 


56.0 


61.5 




Asian or Native American 


5.9 


6.9 


9.2 




Student speaks limited English 3 


8.7 


7.7 


6.2 


Student lives in single-parent household 


49.7 


39.2 


23.8 


* ** 


Neither parent has high school diploma 


26.6 


29.5 


28.8 


Parental Work 








Both parents work 


41.3 


48.3 


51.9 


*** 


Father works 


24.9 


20.6 


29.0 




Mother works 


19.4 


19.8 


12.1 




Neither parent works 


14.3 


11.3 


7.1 




Family receives welfare or Food Stamps 


31.8 


23.0 


18.6 


* ** 


Family mobility in past two years 








Have not moved 


54.2 


60.5 


63.3 


* ** 


Moved 1 or 2 times 


33.5 


34.6 


31.7 




Moved 3 or more times 


12.4 


5.0 


5.0 




Educational characteristics 








8 ll1 -grade math test score 0 








75 th percentile or higher 


3.1 


8.8 


14.0 


* ** 


50 in to 74 w percentile 


16.3 


20.5 


24.8 




25 u ‘ to 49“' percentile 


31.0 


33.9 


29.9 




24 w percentile or lower 


49.5 


36.8 


31.3 
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Table 2.2 (continued) 



Characteristic 


High-Risk 

Subgroup 

(%) 


Medium-Risk 

Subgroup 

(%) 


Low-Risk 

Subgroup 

(%) 




8 th -grade reading test score c 










75 th percentile or higher 


4.1 


10.6 


14.6 


* ** 


50 th to 74 th percentile 


19.7 


19.5 


18.9 




25 th to 49 th percentile 


36.3 


37.6 


33.5 




24 th percentile or lower 


40.0 


32.3 


33.1 




Student does not feel safe at school 


27.0 


22.5 


20.6 


* 


Frequency of cutting classes 










Never 


63.4 


82.2 


89.7 


* ** 


At least 1 time a week 


33.2 


16.9 


10.3 




Daily 


3.4 


0.9 


0.0 




Sent to office for misbehavior 










Never 


73.6 


81.0 


90.4 


* ** 


1 -2 times 


20.7 


16.4 


8.7 




3-10 times 


5.7 


2.5 


1.0 




Characteristics associated with dropping out of school 










Attendance rate, year prior to random assignment 










96-100% 


24.4 


52.5 


91.0 


* * * 


91-95% 


23.6 


32.3 


7.9 




86-90% 


18.7 


11.7 


1.0 




85% or lower 


33.3 


3.6 


0.2 




Credits earned in 9 th grade 0 










5 or more credits 


47.2 


93.8 


100.0 


* * * 


3-4 credits 


35.1 


6.3 


0.0 




2 or fewer credits 


17.7 


0.0 


0.0 




Grade point average in year of random assignment 6 










3.1 or higher 


12.5 


37.2 


58.6 


* ** 


2. 1-3.0 


25.5 


44.0 


39.3 




2.0 or lower 


62.0 


18.7 


2 .1 




Student is overage for grade level 1 


43.0 


18.2 


2.4 


* ** 


School mobility 8 










0 or 1 different school 


50.0 


71.9 


99.0 1 


!••• 


2 or more different schools 


50.0 


28.1 


1.0 J 


1 


Student has sibling who dropped out of high school 


42.7 


17.5 


0.5 


* * * 


Sample size (N= 1,764) 


474 


869 


421 
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SOURCES: See Table 2.1. 



Table 2.2 (continued) 



NOTES: All characteristics were measured at the time students applied to the Career Academy program and prior to 
being randomly selected to the Academy and non-Academy groups. 

Invalid or missing values are not included in individual variable distribution. Rounding may cause slight 
discrepancies in calculating of sums and differences. 

A chi-square test was applied to differences in the distribution of characteristics across the Academy and non- 
Academy groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

‘These are students who responded that they spoke English n not well" or "not at all." 

b Several different standardized, nationally normed math tests were administered to students, depending on the 
district where their school was located and the year they entered the study. National percentile scores were used 
because they were the only standardized scores available across tests. 

c Several different standardized, nationally normed reading tests were administered to students, depending on the 
district where their school was located and the year they entered the study. National percentile scores were used 
because they were the only standardized scores available across tests. 

This was applicable only to students who applied to the Career Academy at the end of their 9 ,h -grade year. 

e Grade point averages were converted to a standard 4.0 scale from 100-point or 5-point scales for some sites. 

f A student is defined as overage for grade at the time of random assignment if she or he turns 1 5 before the start 
of the 9 th grade, or 1 6 before the start of the 1 0 th grade. This indicates that the student was likely to have been held 
back in a previous grade. 

e School mobility is defined as the number of schools attended since the 1 st grade beyond the number expected to 
result from promotions in grade level or graduations. 



subgroup who applied for an Academy at the end of 9 th grade had already fallen behind in the 
number of course credits they needed to graduate. 

Table 2.2 also highlights a number of other distinctive characteristics of students in the 
high-risk subgroup. It shows that these students were the most likely to have S^-grade math or 
reading test scores below the 25 th percentile nationally. About half of them lived in single-parent 
households, and 32 percent reported that their families received public assistance. 

Students Sin the Low-Risk Subgroup. The vast majority of students in the low-risk sub- 
group had attendance rates higher than 95 percent, and all of those who applied to an Academy at 
the end of 9 th grade had earned at least five credits toward graduation. Over half had a grade point 
average of 3.1 or higher, and very few were overage for grade. Almost none of the low-risk stu- 
dents had a sibling who dropped out of high school, and very few had transferred schools other 
than at the typical school transition points. The majority of these students lived in households 
where at least one parent had a high school diploma or GED (over 70 percent), and over half 
lived in households where both parents worked. 

Although students in the low-risk subgroup appeared to be highly engaged in school at 
the time they applied to an Academy, their standardized test scores indicate that they were not 
necessarily high-achieving students. Less than 15 percent of the low-risk subgroup had math or 
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reading test scores above the 75 th percentile nationally, and almost a third scored in the bottom 
quartile. 

Students in the Medium-Risk Subgroup. The students in the medium-risk subgroup 
reflect a mix of background characteristics and prior school experiences. Many of these students 
appeared to be highly engaged in school; the vast majority of those who applied for an Academy 
at the end of 9th grade had earned five or more course credits, and about half had attendance 
rates of higher than 95 percent. At the same time, just under 20 percent were overage for their 
grade level, and just under 20 percent had a grade point average of 2.0 or lower. 

B. Selected Outcomes by Risk Subgroup for the Non-Academy Group 

Figure 2.1 shows three outcomes that are central to the analyses conducted for this report: 
the dropout rate, the percentage of students who completed sufficient course credits to meet their 
districts’ graduation requirements, and the percentage of students who completed a basic core 
academic curriculum (four course credits in English, three course credits in social studies, and 
two course credits each in science and math). Each outcome was measured at the end of the stu- 
dents’ 12 th -grade year. The figure illustrates the percentages of students in the non- Academy 
group from each of the three risk subgroups who attained each of these outcomes. It illustrates 
the dramatic differences among the three subgroups. 

Thirty-two percent of non-Academy students in the high-risk subgroup dropped out of 
high school, and nearly three-quarters had not earned enough credits to graduate from high 
school by the end of their n^-grade year. Only 16 percent of the non-Academy students in this 
subgroup had completed the basic core academic curriculum. The analysis in Chapter 3 assesses 
the extent to which students in the Academy group who had the same background characteristics 
fared better than their non-Academy counterparts. 

Figure 2.1 also illustrates the relatively high level of engagement among non-Academy 
students in the low-risk subgroup. In all, only 3 percent of these students in the non-Academy 
group dropped out of high school, and 75 percent had earned enough credits to meet their dis- 
tricts’ graduation requirements. Just over 60 percent had completed the basic core curriculum. 
The analysis in Chapter 3 assesses the extent to which the Career Academies enhanced or limited 
the capacity of these students to complete their course requirements for graduation and to prepare 
for post-secondary education. 

As expected, the outcome levels for non-Academy students in the medium-risk subgroup 
fell between those in the high- and low-risk subgroups. Eight percent of non-Academy students 
in the medium-risk subgroup dropped out of high school, while 65 percent earned sufficient 
credits to meet district graduation requirements. About half of these students completed the basic 
core curriculum. 



III. Career Academy Enrollment and Attrition Patterns 

This section of the chapter examines the patterns by which students in the study sample 
enrolled and remained in the Career Academies. This information is important to the evaluation 
because the patterns of Academy enrollment and attrition are the key indicators of the extent to 
which students in the sample were exposed to the full range of Academy experiences. Thus, these 
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patterns are key determinants of program impacts and provide a crucial context for interpreting 
the results discussed later in the report. This information is also relevant to policies and practices 
affecting Career Academies because they shed light on the demand for the programs and assess 
their capacity to keep students engaged in their activities and services. 



Figure 2.1 

Career Academies Evaluation 

12th Grade Outcomes Among 
Non-Academy Students, by Risk Subgroup 




SOURCES: MDRC calculations from the Career Academies Evaluation School Records and 12th Grade Survey Databases. 



NOTES: A two-tailed t-test was applied to differences between Academy and non-Academy groups. Statistical significance levels 
are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent. 

indicates completion of four English, three social studies, two math, and two science courses during high school. 



The findings discussed in this section of the chapter are based on 782 students in the 
study sample who were randomly selected for the Academy group at the end of 8 ,h or 9 th grade. 8 
The analysis follows these students through the end of their 12 th -grade year to determine the per- 
centage who actually enrolled in a Career Academy and then examines the patterns by which 
they left or remained in the programs. It also examines the reasons why some of these students 
chose not to enroll in a Career Academy or why they enrolled for a time and then left. 

A. How Many Students Enrolled in a Career Academy and How Many Staved 
in the Programs? 

Figure 2.2 illustrates the enrollment patterns of a typical group of 100 students in the 
study sample selected to enroll in the Academy programs. The numbers in the boxes thus repre- 



8 This includes students in the Academy group from the Career Academies Evaluation Student School Records 
Database. The remaining Academy students in the study sample did not have a complete set of school records and, 
therefore, did not have a complete record of their school enrollment status. 
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sent percentages of the 782 students in the Academy group who were randomly selected to enroll 
in programs. The figure shows that 88 percent of these students enrolled in a Career Academy at 
some point during high school; the vast majority did so during the year following their applica- 
tion to the programs (the first semester of 9 th or 10 th grade). Figure 2.2 also shows that 58 percent 
of the initial group of students were still enrolled in an Academy at the end of their 1 2*-grade 
year. This means that 66 percent of those who were enrolled in an Academy for at least one se- 
mester remained in the programs throughout high school. 

Figure 2.2 also indicates that a significant portion of those initially selected for the pro- 
grams were not exposed to the full range of Career Academy experiences. In all, 42 percent of 
the Academy group either did not enroll in the program or enrolled for a semester or more and 
then left. The next two sections provide an overview of what happened to these students and re- 
view some of the factors that led to their not enrolling or not remaining in the programs. 

B. What Happened to the Students Who Never Enrolled in a Career Academy 
or Who Enrolled and Later Left the Programs? 

The following is a summary of the high school enrollment status of those students who 
either never enrolled in a Career Academy or enrolled and then left the programs before the end 
of high school. 

o 12 percent of the students randomly selected for the Academy group 
never enrolled in a Career Academy. Following is a summary of the high 
school enrollment status at the end of their 12 th -grade year: 

o 4 percent were still enrolled in the high school in which the Academy 
was located (but were not enrolled in the Academy). 

o 5 percent were enrolled in another high school in the same district. 

o 1 percent were enrolled in a high school in another district. 

o 2 percent dropped out of high school. 

o 30 percent of the students randomly selected for the Academy group en- 
rolled in a Career Academy during at least one semester of high school, 
but they left the program before the second semester of their 12 ,h -grade 
year. Nearly 90 percent of these students had left a Career Academy by the 
end of their 1 l^-grade year. This begins an important stage of involvement in 
an Academy because most work-based learning activities occur during the 
summer between 1 1 th and 12 th grades or during the ^-grade year. These stu- 
dents spent an average of three semesters in the program before they left. 
Following is a summary of their school enrollment status at the end of their 
12 th -grade year: 

o 12 percent were still enrolled in the high school in which the Academy 
was located (but were not enrolled in the Academy). 

o 9 percent were enrolled in another high school in the same district. 

o 1 percent were enrolled in a high school in another district. 

o 8 percent had dropped out of high school. 
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Figure 2.2 



Career Academies Evaluation 

Career Academy Enrollment and Attrition Patterns 
Among Students Selected to Enroll 




SOURCE: MDRC calculations from the Career Academies Evaluation Student School Records Database. 



NOTE: Numbers are derived in proportion to 782 students who were selected to enroll in Career Academies. 



In sum, therefore, 90 percent of the students randomly selected for the Academy group 
were still enrolled in high school at the end of their H^-grade year. Following is a summary of 
where they were enrolled in school: 

o 58 percent were enrolled in a Career Academy. 

o 16 percent were enrolled in the high school in which the Academy was located. 

o 15 percent were enrolled in another high school in the same district. 

o 1 percent were enrolled in a high school in another district. 

Also, by the end of their H^-grade year, approximately 10 percent of the students in the Acad- 
emy group had dropped out of high school. 

C. Wlhv Did Students Not Enroll in a Career Academy or Enroll and Then Leave? 

Of the students who applied for a Career Academy and were selected to enroll, 42 percent 
had either never enrolled in an Academy or had enrolled and then left before the second semester 
of their H^-grade year. A subset of 251 completed the 12th Grade Survey and provided infor- 
mation about why they did not enroll in an Academy or why they enrolled and then left. Students 
were asked to rate up to 16 items in terms of how important each was as a reason for not enroll- 
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ing in an Academy or enrolling and then leaving. For the purposes of this analysis, the primary 
reasons for never enrolling, or for enrolling and then leaving, were divided into four mutually 
exclusive groups: student choice, family mobility and school transfer, being asked to leave, and 
dropping out. 9 

The following is a summary of the reasons that students listed as the most important fac- 
tors that led them to never enroll in a Career Academy or to enroll for at least one semester and 
then leave before the end of their ^-grade year. 

• Student choice. 54 percent of the students reported one or more reasons, indi- 
cating that they chose not to enroll or chose to leave the Academy. Among 
those who chose to leave or not to enroll, the most common reasons in- 
cluded: 10 

• I wanted to enroll in another program. (42 percent) 

• I was not really interested in the program to begin with. (39 percent) 

• I did not think the program would help me get into the college I 
wanted. (32 percent) 

• I did not like the teachers. (36 percent) 

• I lost interest in the occupational area. (35 percent) 

• Mobility. 23 percent of the students reported that they moved and had to 
transfer to another high school. 

• Being asked to leave. 16 percent reported that they were asked to leave the 
Academy. 

® Dropping out. 5 percent reported that they stopped going to high school. 

® No primary reason. 2 percent did not indicate a primary or secondary reason. 

This suggests that most of the attrition from the Career Academies is the result of a deci- 
sion on the part of students. However, nearly one-quarter of the attrition appears to be a function 
of family mobility and school transfers. 



9 Students were asked to rate each reason on a 4-point scale: 1 = very important, 2 = sort of important, 3 = not 
very important, and 4 = not important at all. Items that were rated as “very important” were designated as primary 
reasons, and items rated as “sort of important” were designated as secondary reasons. If a student did not list a pri- 
mary reason, then the secondary reason was designated as the primary reason. 

l0 These are not mutually exclusive categories. Many students listed several reasons as being “very important” 
or “sort of important.” In addition to the reasons listed, other reasons students chose to leave or not enroll in the 
Academies included: “The program was too hard”; “I wanted to be in classes with more of my friends”; “I was tired 
of being in classes with the same students”; “I did not think the program would help me get the job I wanted”; and 
“I chose to leave for other reasons.” 
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B. Who Is Most Likely to Enroll and Remain in a Career Academy? 

Table 2.3 presents the Career Academy enrollment and attrition rates for selected sub- 
groups of students who were randomly selected for the Academy group. The table shows the per- 
centage of each subgroup who had enrolled in an Academy at some point during high school and 
the percentage who remained enrolled in the programs through the end of 12 th grade. The table 
also shows the percentage of each subgroup who enrolled during at least one semester of high 
school but who then left before the end of 12 th grade. Finally, the last column of the table shows 
the average number of semesters that students in each subgroup were enrolled in an Academy. 

In general, the table indicates that there were some modest differences among various 
subgroups in the percentage of students who had enrolled in an Academy during at least one se- 
mester during high school. Academy enrollment rates ranged from 80 to 95 percent for most of 
the subgroups. It is interesting to note that the initial Career Academy enrollment rates were quite 
similar among the three risk subgroups. 

Table 2.3 does indicate somewhat more variation among the various subgroups in the 
percentage of students who remained in a Career Academy through the end of their 12 th -grade 
year. In general, students in the high-risk subgroup were less likely than medium- and low-risk 
students to be enrolled in a Career Academy through the end of 12 th grade. This can be seen in 
the last section of the table, which shows the enrollment rates for the risk subgroups. It shows 
that 43 percent of students in the high-risk subgroup were enrolled in an Academy at the end of 
12 th grade, indicating that about half of those who initially enrolled eventually left the programs. 
On average, these students spent just under four semesters in a Career Academy. For the majority 
of high-risk students, therefore, the benefits that may have derived from the Academies were 
likely to occur during 10* and 11* grades (or 9* and 10* grades in sites where the Academies 
began in 9* grade). Thus, many students in the high-risk subgroup did not stay in the programs 
long enough to participate in the work-based learning programs and work internships, which 
typically occur after 1 1* grade. 

By contrast, 73 percent of the students in the low-risk subgroup were enrolled in a Career 
Academy at the end of 12* grade. This means that over 80 percent of these students who initially 
enrolled in a Career Academy remained enrolled in the programs throughout high school. On av- 
erage, students in the low-risk subgroup spent over five semesters in a Career Academy. 11 

E. Implications for Career Academy Implementation and for the 
Career Academies Evaluation 

The enrollment and attrition patterns discussed above have implications for policy and 
practice related to the Career Academies. They also provide an important context for interpreting 
the impact findings discussed in Chapter 3. These are discussed briefly below. 

Implications for Career Academy Policies and Implementation. It is unclear how 
much of the attrition could theoretically be controlled or avoided by the Academies. Student mo- 



"Note that for approximately 80 percent of the students in the sample, the Career Academies began in 10 ,h 
grade. These students had the opportunity to remain enrolled in an Academy for up to six semesters. For the re- 
maining students, the Career Academies began in 9 th grade, providing students with the opportunity to enroll for up 
to eight semesters. 
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Both parents work 339 88.5 60.8 27.7 

Father works 171 92.4 61.4 31.0 

Mother works 139 87.8 59.0 28.8 

Neither parent works 79 78.5 45.6 32.9 
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bility and dropout are not uncommon problems in most urban high school districts, and they 
were factors associated with the attrition from the Career Academies in this evaluation. As dis- 
cussed earlier, nearly one-quarter of the students who never enrolled or left the Academies did so 
because they moved. This finding suggests that at least part of the attrition from the Academies 
results more from family relocation patterns than from any particular feature or shortcoming of 
the Academy (although some families may have moved to find better schools). 

However, just over half of those who never enrolled or who enrolled and then left indi- 
cated that they chose to do so. It is not unreasonable to expect that a substantial number of the 
Academy students who enroll in 9 th or 10 th grade would decide that the Academy program is not 
the best context in which to pursue their education. In some cases students may leave the Acad- 
emy because they decide that they are no longer interested in the career theme or in the various 
work-related learning activities. This attrition need not imply shortcomings in the Academy 
model or its implementation if the students made the choice to leave because they had access to 
opportunities that better suited their needs and interests. 

Many high schools and school districts around the country are attempting to expand the 
number of Career Academies they operate. In some cases, there are efforts to convert entire high 
schools to a series of Career Academies (often referred to as “wall-to-wall” Academies). In these 
cases, every student in the school would be required to enroll in an Academy beginning in 10 th 
grade (or, in some cases, in 9 th grade). The findings on enrollment and attrition from this study 
suggest that there may not be an excess of demand for Academies, at least under circumstances 
where students have the opportunity to choose to leave and staff have the opportunity to ask stu- 
dents to leave. 

Each of the programs in the study received applications from more students than it was 
able to serve. This suggests that there was likely to be enough demand for Career Academies to 
justify expanding the number of programs within the high schools. However, given the opportu- 
nity to choose to leave the programs — or, on the part of staff, given the opportunity to dismiss 
students from the programs — less than 60 percent of the students remained in the programs 
throughout high school. This suggests that the current programs may be operating at capacity, 
based on the number of students who choose to stay and whose mobility patterns permit them to 
do so. 



Interpreting the Impact Findings. The fundamental comparison being made in the im- 
pact analysis is between outcomes for students who were selected to enroll in an Academy (the 
Academy group) and outcomes for students who were not selected (the non-Academy group). 
Differences between the groups were driven by the extent to which the Academy group was ex- 
posed to the Academy experiences and the extent to which the non-Academy group was not. The 
analysis in this chapter shows, however, that not all the students randomly assigned to the Acad- 
emy group actually enrolled in an Academy. In addition, approximately 6 percent of the students 
in the non-Academy group were inadvertently allowed to enroll in Academies. Thus, differences 
between the two groups reflect the Academy-related experiences of those students in the Acad- 
emy group who enrolled in an Academy (88 percent) and the experiences of those students in the 
non-Academy group who did not enroll in an Academy (94 percent). 

In order to produce unbiased estimates of the Academies’ true impact, it is necessary to 
include all sample members in the analysis, regardless of their Academy enrollment status. For 
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example, students who dropped out of high school are considered to have attended school for 
zero days and to have earned zero credits during the period they had left school. To the extent 
that the Career Academies keep students in their programs and prevent them from dropping out, 
excluding these zero values from the analysis would lead to a serious underestimation of the pro- 
gram impacts. At the same time, the findings in the chapter show that students in the high-risk 
subgroup are more likely to leave the Academies. If these students were excluded from the 
Academy group but not from the non-Academy group, it would appear that the Academies in- 
clude fewer low-achieving and less engaged students. This would represent a serious overesti- 
mation of the impact of Career Academies. 

Nevertheless, it is highly unlikely that the Career Academies had much effect on students 
in the study’s Academy group who never enrolled in them. It is also not accurate to assume that 
the Academies had no effect on students in the study’s non-Academy group who did enroll. From 
this perspective, the impact estimates may be perceived as being “diluted.” Thus, it is useful to 
provide impact estimates that account for these “crossovers” in research status. 

In the case of the Career Academies Evaluation, the impacts are adjusted based on the 
difference in actual enrollment rates among Academy and non-Academy groups. In particular, 
each of the tables in Chapters 3 and 5 includes the impact per enrollee for each outcome. This is 
defined as the difference between the outcomes of Academy and non-Academy students divided 
by the difference between the percentage of Academy and non-Academy students who ever en- 
rolled in an Academy. This adjustment is discussed further in Chapter 3. 

These adjustments do not substantially change the overall impact story discussed in 
Chapters 3 and 5. The impact per enrollee can be interpreted as the impact from actually enroll- 
ing in an Academy, as opposed to simply being recruited and selected for admission. Chapters 3 
and 5 explore the extent to which the patterns of the impacts of Career Academies on student out- 
comes vary across different subgroups of students. The analysis presented above suggests that these 
impacts are not greatly affected by differences across the subgroups in the extent to which these 
students ever enroll in an Academy. However, it suggests the possibility that impacts are affected 
by differences in how long subgroups remain in an Academy. 
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Chapter 3 



Career Academy Impacts on 
Stindemt Engagement, Performance, and Achievement 



Chapter 2 described the background characteristics and prior school experiences of the 
students who are the focus of this report. This chapter evaluates the impact of Career Academies 
on a broad range of high school outcomes for students at different levels of risk of school failure. 

The random assignment research design used in this study provides a uniquely rigorous 
way to identify the impact of Career Academies. The students in this study were randomly as- 
signed to either the Academy group, which had access to an Academy, or to the non-Academy 
group, which did not. As a result, the impact of the Career Academies is defined as the difference 
in the outcome levels achieved by Academy students over and above those achieved by their 
non-Academy counterparts. These impact findings provide the most reliable estimates of the true 
difference these Academies made for the students they aimed to serve. 

Most previous evaluations of Career Academies have tended to focus on program out- 
comes and impacts for the “typical” or “average” Career Academy student. Like these previous 
studies, this chapter provides a brief summary of results that are averaged across the diverse 
group of students and sites participating in this evaluation. However, as discussed in Chapter 2, it 
is clear from both a substantive and a statistical standpoint that these “average” results mask the 
high degree of underlying variation in impacts. In particular, the analysis in Chapter 2 revealed 
that, within this study sample, there are groups of students who had substantially different back- 
ground characteristics and who, in the absence of the Academy treatment, experienced substan- 
tially different academic outcomes at the end of high school. 

Therefore, to adequately understand the impact of these Career Academies, it is impor- 
tant to recognize that they affect students differently depending on the types of skills, attributes, 
and prior experiences that the students bring to the programs. Thus, the impact findings dis- 
cussed in this chapter are presented separately for the three subgroups of students described in 
Chapter 2: students at high risk of dropping out and highly likely to be disengaged if they stayed 
in school (approximately 25 percent of the study sample); students at low risk of dropping out 
and likely to remain engaged and perform well in school (approximately 25 percent of the study 
sample); and students at only medium risk of dropping out but not necessarily highly engaged in 
school (approximately 50 percent of the study sample). 1 

The evidence presented in this chapter suggests the following conclusions: 

o Among students most at risk of dropping out, Career Academies significantly 
improved high school outcomes. The Academies reduced dropout rates, im- 
proved attendance, increased academic course-taking, and increased the likeli- 
hood that students graduated on time. 

'Each of these subgroups comprises students from the study’s Academy and non-Academy groups as deter- 
mined at random assignment. As discussed in Appendix B, there are no systematic differences in measured back- 
ground characteristics between Academy and non-Academy students within each subgroup. 
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® Among the students least likely to drop out of high school, Career Academies 
increased the likelihood that students were prepared to graduate on time. Be- 
cause both Academy and non-Academy students in the low-risk subgroup 
were likely to remain strongly engaged in high school, the Academies had lit- 
tle or no impact on most indicators of student engagement and performance. 

• On average, the Career Academies produced little or no change in outcomes 
for students in the medium-risk subgroup. 

• When data are averaged across the diverse groups of students and sites par- 
ticipating in the evaluation, it appears that the Career Academies produced 
only modest improvements in students’ engagement and performance during 
high school. 

To provide a context for interpreting the estimates presented in this chapter, Section I 
briefly reviews several important analysis issues. 



I. Analysis Issues 

When examining the effectiveness of Career Academies in influencing students’ behavior 
and experiences, it is important to distinguish between measures of program “outcomes” and 
measures of program “impacts.” Outcomes refers to the measures of student engagement, per- 
formance, behaviors, achievement, and attitudes — in this case, during their high school years. 
The chapter examines five sets of outcomes that were measured through the end of each student’s 
12 th -grade year: 

• high school enrollment and attendance rates; 

• credits earned and course-taking patterns; 

• math and reading achievement test scores; 

• use of non-school hours and involvement in negative risk-taking behaviors; 
and 

• steps taken toward further education and work and plans for the future. 

As noted in Chapter 1, these outcomes were measured using data collected from school tran- 
script records, a survey that students completed at the end of their Upgrade year, and a math 
and reading achievement test that a subsample of students completed at the end of their Up- 
grade year. 

An impact is defined as the effect that a Career Academy has on an outcome. The average 
outcome levels for students in the Academy group alone provide potentially misleading conclu- 
sions. Previous research and prior experience highlight the fact that many students succeed or fail 
in high school for reasons not related to a special intervention like a Career Academy. In order to 
determine the net effect, or value added,” of a Career Academy, it is necessary to compare the 
experiences of a group of students who were exposed to a Career Academy with a similar group 
of students who also applied but were not selected to enroll. As discussed in Chapter 1 , the 
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Academy and non-Academy groups participating in this study were determined through a ran- 
dom selection process. The non-Academy group serves as a benchmark for how students in the 
Academy group would have performed if they had not had access to the programs. Therefore, the 
impacts (differences in outcomes between the Academy and the non-Academy groups) represent 
the difference in outcomes that Career Academies generate over and above what non-Academy 
environments do for comparable students. 

Unless otherwise noted, the measures presented in the tables for this chapter indicate 
the percentages of students in the Academy and non-Academy groups who attained a given 
outcome or reported a given behavior or experience. For example, some tables report the per- 
centages of students who dropped out of high school, who attended 95 percent or more of the 
time throughout high school, or who earned a sufficient number of course credits to meet the 
school district’s graduation requirements. Other tables report the percentages of students who 
reported working on volunteer projects, who reported being arrested, or who reported submit- 
ting a college application. 

Each table compares the percentage of Academy group students who attained a given 
outcome with the percentage of non-Academy group students who did so. The difference be- 
tween the groups represents the impact of the Career Academies. The tables also present the per- 
centage change in the non-Academy group outcome level represented by the impact. This is de- 
fined as the impact divided by the non-Academy group average. For example, if 60 percent of the 
Academy group attained a certain outcome compared with 50 percent of the non-Academy 
group, this 10 percentage point difference would represent a 20 percent increase (10 divided by 
50) over the non-Academy group level of 50 percent. 

It is important to note that the impact estimates discussed in this report are based on 
analyses that include all students in both the Academy and the non-Academy groups. This 
includes both Academy group students who may not have enrolled in a Career Academy and 
students from both groups who may have dropped out of high school altogether. As discussed in 
Chapter 2, not all students randomly selected for the Academy group actually enrolled in an 
Academy. To the extent that these students are different from their counterparts who did enroll 
(and, more important, different from students in the non-Academy group), excluding them from 
the analyses would lead to serious misrepresentation of the impacts. 

At the same time, it is unlikely that the Academies had any effect on students who were 
never involved with the program after they were selected to enroll. In an effort to account for 
this, each table presents the impact per enrollee, defined as the estimated impact divided by the 
difference in actual Academy enrollment rates of Academy and non-Academy students. In other 
words, the impact per enrollee can be interpreted as the impact from actually enrolling in an 
Academy, as opposed to simply being recruited and selected for admission. 2 



2 This adjustment was proposed by Bloom, 1 984, and was used by Orr, Bloom, Bell, Doolittle, Lin, and Cave, 
1996. It relies on two assumptions: (1) selection for the Academy group had no effect on students who did not enroll 
in an Academy and (2) the average outcome levels for non-Academy students who did enroll would have been the 
same if they had been assigned to the Academy group initially. Thus, the adjustment can be seen as discounting 
both the zero impact for that portion of the Academy group who did not receive any part of the Academy treatment 
and the non-zero impact for that portion of the non-Academy group who got the same treatment as the Academy 

(continued) 
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It is also important to note that high school dropouts from both the Academy and the 
non-Academy groups are included in calculations of outcomes. For example, estimates of av- 
erage attendance rates or credits earned toward graduation include zero values for school years 
or semesters in which students were confirmed to have dropped out of high school. To the ex- 
tent that Career Academies prevent Academy students from dropping out, excluding non- 
Academy group students with zero values (that is, dropouts) would lead to serious underesti- 
mation of program impacts. 

Finally, another issue of interpretation concerns the “statistical significance” of impact 
estimates. Statistical significance is a measure of the degree of certainty one may have that some 
non-zero impact actually occurred. If an impact estimate is statistically significant, then one may 
conclude with some confidence that the program really had an effect. If an impact estimate is not 
statistically significant, then the non-zero estimate is more likely to be the product of chance. 
Unless otherwise noted, the impacts discussed in this chapter were statistically significant at the 
10 percent level or lower. This means that there is no more that a 10 percent probability that the 
difference resulted only from chance. 

Statistical significance does not directly indicate the magnitude or importance of an 
impact estimate — only whether any impact occurred. In an evaluation such as this one, 
numerically small impact estimates are usually not statistically significant; however, some 
numerically large impact estimates may not be statistically significant, particularly when sample 
sizes are small. Smaller sample sizes yield less reliable impact estimates — estimates in which 
one can have less confidence — than are possible when samples are larger. Therefore, an 
estimate of a given magnitude that is statistically significant for a relatively large subgroup may 
not be statistically significant for a smaller subgroup. 



II. Career Academy Impacts for Stmdemits an the High-Risk Smtogromip 

This section of the chapter focuses on those students in the study sample who were most 
likely to drop out of high school or to perform poorly if they stayed in school. As discussed in 
Chapter 2, this subgroup of students was identified based on background characteristics and 
school experiences prior to their applying for an Academy program. The high-risk subgroup rep- 
resents approximately 25 percent of the study sample. The subsequent behavior and performance 
of students in the non-Academy group provide the best indication of how these students per- 
formed in high school without the opportunity to attend an Academy. 

In general, non-Academy students in the high-risk subgroup dropped out of high school 
at relatively high rates and appeared to be quite disengaged if they stayed. In all, nearly one-third 
of these students dropped out of high school before the end of their 12 th -grade year, and ap- 
proximately one-quarter had earned sufficient credits to meet their districts’ graduation require- 
ments. Sixteen percent of the high-risk non-Academy group had completed what might be called 
a basic core academic curriculum (four credits of English, three credits of social studies, two 
credits of math, and two credits of science). On average, these students scored below the 20 th per- 



enrollees. This adjustment does not account for the fact that some students enrolled in an Academy for a semester or 
more and then left. Further analysis is needed to explore the impact of different “doses” of Academy treatment. 
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centile nationally on a standardized math and reading achievement test, and just over 20 percent 
reported that they had taken the SATs or ACTs by the end of their 12 th grade year. 

The findings discussed in this section of the chapter indicate that the Career Academies 
significantly improved high school outcomes among students in the high-risk subgroup. For 
these students, the Career Academies substantially reduced dropout rates and chronic absentee- 
ism, and they improved attendance, credits earned, course-taking patterns, and preparation for 
post-secondary educational and employment opportunities. Despite these significant positive im- 
pacts, the Academies did not produce changes in the high-risk subgroup’s math or reading 
achievement test scores. Before discussing these findings in detail, it is useful to review the rates 
at which these students enrolled in and remained in the Academy programs. 

Table 3.1 provides a summary of Career Academy impacts on the school enrollment 
status and attendance rates of students in the high-risk subgroup. The first row of the table shows 
the percentage of Academy and non-Academy group students who enrolled in a Career Acad- 
emy. The difference between these percentages represents the basic difference in exposure to the 
Career Academies between these groups. This is the primary source of impacts that are discussed 
in this section of the chapter. As noted above, in addition to the basic impact estimates, each ta- 
ble presents the impact per enrollee. 

The second row of Table 3.1 shows the percentage of students in the high-risk subgroup 
who remained enrolled in an Academy through the end of their 12 th -grade year. It indicates that 
43 percent of the Academy students in the high-risk subgroup did so — just under half of those 
who initially enrolled. This may be perceived as a high retention rate in the Career Academies, 
given that many of the students in the high-risk subgroup would have dropped out of high school 
altogether, if they did not have access to an Academy. Also, as discussed in Chapter 2, a rela- 
tively high proportion of the high-risk subgroup came from families with a history of relatively 
high mobility. 

A. Impacts on School Enrollment and Attendance 

The third row of Table 3.1 indicates that the Academies significantly reduced the per- 
centage of students in the high-risk subgroup who dropped out of high school. Among those in 
the non-Academy group, 32 percent dropped out of high school before the end of 12 th grade. In 
other words, without access to an Academy, nearly one-third of the high-risk subgroup left 
high school before they were scheduled to graduate. By comparison, 21 percent of the high- 
risk Academy group had dropped out before the end of their 1 2 th -grade year. This 1 1 percent- 
age point difference represents a 34 percent reduction in dropout rates among students in the 
high-risk subgroup. 

Rows 4 and 6 of Table 3.1 shows that the Academies also improved attendance rates, 
particularly by reducing chronic absenteeism (defined as attending class less than 85 percent of 
the time throughout high school). The non-Academy students in the high-risk subgroup, without 
the opportunity to attend an Academy, had average attendance rates of 76 percent. Over half of 
these students were chronically absent, but the Academies significantly increased average atten- 
dance, from 76 to 82 percent. They also reduced chronic absenteeism, from 53 to 42 percent. 
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Table 3.1 

Career Academies Evaluation 



Impacts on School Enrollment and Attendance 
for Students in the High-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 3 


Impact per 
Enrollee b 


Ever enrolled in a Career Academy 
during high school (%) 


86.4 


3.2 


83.3*** 






Was enrolled in a Career Academy 
at the end of grade 12 (%) 


42.5 


2.1 


40.5 *** 






Dropped out of high school 
before the end of grade 12 (%) 


21.3 


32.2 


-10.9 *** 


-33.8 


- 13.1 


Average attendance, 
grades 9-12 (%) 


81.5 


76.0 


5.6*** 


7.3 


6.7 


More than 95 percent average 
attendance, grades 9-12 (%) 


16.9 


12.9 


4.0 


31.3 


4.8 


Less than 85 percent average 
attendance, grades 9-12 (%) 


41.9 


53.4 


-115 *** 


-21.5 


- 13.8 


Sample size (N=345) 


185 


160 









SOURCE: MDRC calculations from Career Academies Evaluation Student School Records Database. 



NOTES: Attendance rates include zero values for grades in which sample members were identified as 
school dropouts. Estimates are regression-adjusted using ordinary least squares, controlling for 
background characteristics of sample members. Rounding may cause slight discrepancies in 
calculating differences. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. 

In both cases, statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

a Percent change is defined as the impact divided by the non-Academy group average. 
b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy 
and non-Academy group members ever enrolled in a Career Academy. It is italicized because its 
calculation does not involve a direct comparison of Academy and non-Academy students. 



B. Impacts on Credits Earned and Course-Taking 

Table 3.2 presents several outcome measures that indicate the distribution of course cred- 
its earned by Academy and non-Academy students in the high-risk subgroup. 3 The table indicates 



3 Throughout this report, one credit is equivalent to completing one full-year course, and a half-credit is equiva- 
lent to completing one semester-long course. In general, the grouping of courses into subject areas (such as English, 
math, vocational/career-related, and so on) and subject types (academic and non-academic) follows conventions 
outlined in NCES, 1995. 



ERjt 
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Table 3.2 

Career Academies Evaluation 



Impacts on Credits Earned and Course-Taking 
for Students in the High-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 3 


Impact per 
Enrollee 0 


Credits earned 












Total course credits 


19.3 


17,3 


2.0 *** 


11.7 


2.4 


Total course credits meet the 
graduation requirement (%) 


39.9 


26.2 


13.7 *** 


52.3 


16.5 


Earned 1 2 or more academic 
course credits (%) 


47.6 


31.9 


15.8 *** 


49.5 


19.0 


Earned 8 or more non-academic 
course credits (%) 


46.4 


47.2 


-0.8 


-1.8 


- 1.0 


Course-taking 












English (4), Social Studies (3), 
Math (3), Science (3) c (%) 


13.9 


5.7 


8.2 


145.4 


9.9 


English (4), Social Studies (3), 
Math (2), Science (2) c (%) 


31.8 


16.3 


15.5 *** 


94.8 


1 8.6 


Earned 2 or more foreign-language 
credits (%) 


26.7 


19.1 


7.5 


39.4 


9.1 


Earned 1/2 or more computer 
credits (%) 


43.4 


45.0 


-1.6 


-3.6 


- 2.0 


Earned 3 or more career/vocational 
credits (%) 


58.3 


37.7 


20.6 *** 


54.5 


24.7 


Sample size (N=345) 


185 


160 









SOURCE: See Table 3.1. 



NOTES: Credits include zero values for grades in which sample members were identified as school 
dropouts. Course credit data were not provided for approximately 4.5 percent of the sample. Course-taking 
data were not provided for approximately 21 percentof the sample. Estimates are regression-adjusted using 
ordinary least squares, controlling for background characteristics of sample members. Rounding may cause 
slight discrepancies in calculating differences. All measures indicate credits earned up until the end of the 
1 2 th -grade year. 1 2 th -grade year indicates the year that students were projected to reach the 1 2 th grade when 
they initially enrolled in the Career Academy or regular high school program. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In 
both cases, statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
a Percent change is defined as the impact divided by the non-Academy group average. 
b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy 
and non-Academy group members ever enrolled in a Career Academy. It is italicized because its 
calculation does not involve a direct comparison of Academy and non-Academy students. 
c Numbers refer to the amount of credits that were earned in each subject area. 



o 
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that Career Academies increased total course credits earned by high-risk students and increased 
the percentage of students who earned sufficient course credits to meet their districts’ graduation 
requirements. Moreover, much of the increase in total course credits came from an increase in the 
number of academic course credits earned. 

The first row of Table 3.2 reports the average number of course credits earned by Acad- 
emy and non-Academy students in the high-risk subgroup by the end of their scheduled 12 th - 
grade year. 4 It indicates that the Academies produced an increase of two full credits. While non- 
Academy students earned an average of approximately 17 credits by the end of 12 th grade, their 
Academy counterparts earned an average of approximately 19 credits. This difference represents 
an increase of 12 percent over the non-Academy group level. 

The second row of Table 3.2 reports the percentage of students in the Academy and non- 
Academy groups who earned enough credits to meet their districts’ graduation requirements by 
the end of their 12 th -grade year. 5 It shows that 26 percent of the high-risk non-Academy group 
earned enough credits to graduate. By contrast, nearly 40 percent of the Academy group students 
earned enough credits to graduate. This 14 percentage point difference represents a 52 percent 
increase over the non-Academy group average. 

The remainder of Table 3.2 presents findings on the credits earned in various subject ar- 
eas by Academy and non-Academy students in the high-risk subgroup. Career Academies sig- 
nificantly increased the number of academic courses students completed, as well as the number 
of career-related and vocational courses. 6 The third row of the table indicates that the Academies 
significantly increased the percentage of students in the high-risk subgroup who completed 12 or 
more academic credits (a minimum of three per year). While 32 percent of the non-Academy 
group earned 12 or more credits in academic courses, 48 percent of students in the Academy 
group did so. This 16 percentage point difference represents a 50 percent increase over the non- 
Academy group average. 

This overall increase in academic course credits translated into a substantial increase in 
the percentage of students who completed a core academic curriculum. The fifth and sixth rows 
of Table 3.2 indicate the percentages of Academy and non-Academy students from the high-risk 
subgroup who completed two versions of an academic curriculum that prepared them for col- 
lege. 7 The second measure (row 6) can be classified as a basic academic core curriculum con- 
sisting of four credits of English, three of social studies, two of math, and two of science. The 
first measure (row 5) might be classified as a more intensive core curriculum that adds an extra 
credit in both math and science. Many school districts require student to complete this type of 



4 When available, this measure includes credits that students earned during summer school. Some of the partici- 
pating school districts were unable to provide a complete record of summer school credits. 

5 Students were considered to have earned enough credits to graduate from high school if their transcripts indi- 
cated that they had accumulated the number of credits needed to meet the official graduation requirements in their 
school district. This ranges from 21 to 24 credits, depending on the district. It is also important to note that this 
measure does not necessarily indicate that a student actually graduated from high school on time. Most school dis- 
tricts also require students to complete a certain number of courses in specific subject areas. 

6 Academic courses included those in English, social studies, math, science, and foreign language. Non- 
academic courses were all other courses, including career-related and vocational subjects, accredited work experi- 
ences, fine arts, physical education and health, and a broad range of school-specific electives. 

’These measures have been proposed by the National Center for Education Statistics; see NCES, 1995. 
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course distribution in order to graduate, and many four-year colleges require that students com- 
plete these types of courses for admission. 

Table 3.2 indicates that very few non- Academy students in the high-risk subgroup 
completed enough academic courses to meet requirements for either of the core academic cur- 
ricula. Sixteen percent of non-Academy students completed the basic academic curriculum, 
and less than 6 percent completed the more intensive version. In all, Career Academies nearly 
doubled the percentage of students in the high-risk subgroup who completed the basic core 
academic curriculum, from 16 percent of the non-Academy group to nearly 32 percent of the 
Academy group. Although the Academy group students were much more likely than their non- 
Academy counterparts to complete the more intensive core curriculum, this difference was not 
statistically significant. 

Finally, Table 3.2 shows that although the Academies did not increase the overall number 
of credits that the high-risk subgroup earned in non-academic subject areas, they substantially 
increased credits earned in career-related and vocational courses. The fourth row of the table in- 
dicates that Academy and non-Academy students were about equally likely to complete eight or 
more credits in non-academic subject areas. As shown in the last row of the table, however, the 
Academies increased the percentage of students completing three or more career-related and vo- 
cational courses from approximately 38 percent for the non-Academy group to 58 percent for the 
Academy group — an increase of nearly 55 percent over the non-Academy group average. 

In summary, this combination of findings is significant for several reasons. First, many 
students in the high-risk subgroup were already lagging behind in credits at the time they entered 
an Academy. Thus, the Academies not only prevented students from dropping out but also 
helped a number of students close their initial gap in credits and meet the graduation require- 
ments. Second, the Academies’ positive impact on credits earned resulted primarily from an in- 
crease in academic course-taking and there was no overall increase in non-academic course- 
taking. In fact, students in the high-risk subgroup were more likely to concentrate their elective 
and non-academic courses in career-related or vocational subject areas rather than to substitute 
non-academic courses for academic courses. 

C. Impacts on Math and Reading Achievement Test Scores 

The evidence presented so far suggests that, among students at high risk of school fail- 
ure, Career Academies reduced dropout rates and increased engagement in school. The evi- 
dence presented below suggests that, despite the effects of Career Academies on these out- 
comes, they appear to have little or no effect on standardized measures of student achievement 
in math and reading. 

Table 3.3 presents estimates of the differences in achievement test scores between Acad- 
emy and non-Academy students from the high-risk subgroup. The first row of the table presents 
the students’ average percentile scores on the mathematics portion of the achievement test from 
the National Educational Longitudinal Study of 1988 (NELS: 88). These percentile scores reflect 
the average performance of students relative to the sample of 17- to 1 8-year-olds who made up 
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Table 3.3 

Career Academies Evaluation 



Impacts on Achievement Test Scores 
for Students in the High-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 3 


Impact per 
Enrollee b 


Math achievement test scores 


Average national percentile 


19.5 


16.1 


3.5 


21.7 


4.2 


Proficiency level (%) 


1 : Rote memory operations 


74.1 


62.5 


11.6 


18.5 


13.9 


3: Simple problem-solving 


11.0 


12.1 


-1.2 


-9.6 


- 1.4 


Reading achievement test scores 


Average national percentile 


23.4 


18.9 


4.6 


24.2 


5.5 


Proficiency level (%) 


1: Simple comprehension 


69.2 


70.8 


-1.6 


-2.3 


- 1.9 


2: Simple inferences 


27.6 


17.3 


10.3 


59.8 


12.4 


Sample size (N=l 10) 


63 


47 









SOURCE: MDRC calculations from Career Academies Evaluation 12th Grade Achievement Test 
Database. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for background 
characteristics of sample members. Rounding may cause slight discrepancies in calculating differences. 
The reading and math achievement tests are the cognitive battery tests of reading and mathematics used 
in the NELS: 88 study. There were a total of five proficiency levels for mathematics and three for 
reading. Particular proficiency levels are reported in the table to illustrate general trends in performance 
in the distribution of students. Percentile scores reflect students' performance in relation to a nationally 
representative sample of 12 th -graders. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In 
both cases, statistical significance levels are indicated as: *** - 1 percent; ** = 5 percent; * = 10 
percent. 

a Percent change is defined as the impact divided by the non-Academy group average. 
b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy 
and non-Academy group members ever enrolled in a Career Academy. It is italicized because its 
calculation does not involve a direct comparison of Academy and non-Academy students. 



the original NELS: 88 sample. 8 The table indicates that, on average, the non-Academy students 
in the high-risk subgroup scored at the 16 th percentile on the math achievement test and at the 19 th 



8 The NELS: 88 sample comprised a nationally representative group of students who were identified as 8 th - 
graders in 1988. These young people were surveyed every two years through 1992, when they were scheduled to 
graduate from high school, and again in 1994, approximately two years after scheduled graduation. The achieve- 
ment test being used in the Career Academies Evaluation was administered to the NELS: 88 sample in 1992, at the 
end of their 12 th -grade year. 
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percentile on the reading achievement test. Although students in the high-risk Academy group 
scored somewhat higher, the differences were not statistically significant. 

The second and third rows of Table 3.3 present the percentages of Academy and non- 
Academy students in the high-risk subgroup who, on the basis of their test scores, exhibited pro- 
ficiency at each of two different levels of math skills. Overall, the testing instrument covers five 
levels of math proficiency. Level 1 represents the most basic skills, and level 5 represents the 
highest. Level 1, or basic proficiency, includes the abilities to perform simple rote memory op- 
erations and to carry out simple arithmetical operations on whole numbers. Level 3 reflects a 
somewhat more advanced level of proficiency, including the ability to perform simple problem- 
solving. 9 Table 3.3 indicates that the Academy students were somewhat more likely to attain the 
basic proficiency level than their non-Academy counterparts. Again, however, this difference is 
not statistically significant, indicating that there was no systematic difference between the Acad- 
emy and non-Academy students on this measure. The table also indicates that very few of the 
high-risk subgroup attained the third level of proficiency and that the Academy and non- 
Academy students were about equally likely to do so. 

The results for reading achievement test scores show a similar pattern. The average per- 
centile scores on the reading portion of the NELS: 88 achievement test indicate that non- 
Academy students scored at about the bottom fifth of the national distribution. Although the 
Academy group scored somewhat higher, the difference was not statistically significant. The 
reading test included three levels of proficiency. Approximately 70 percent of students in the 
Academy and non-Academy groups scored at level 1, demonstrating basic reading comprehen- 
sion skills. Students in the Academy group were somewhat more likely than their counterparts in 
the non-Academy group to attain level 2 reading proficiency, indicating that they were able to 
make “simple inferences” from a reading passage. Again, however, these differences were not 
statistically significant. 

B. Im pacts on Yoiith Development Experiences 

Beyond their effect on student performance and engagement in school, Career Academies 
are intended to improve a variety of outcomes that have been identified as important to youth de- 
velopment more generally. In particular, Career Academies aim to increase students’ contact 
with caring adults and to help them make constructive use of non-school hours. To the extent that 
this occurs, one might expect the high-risk Academy group to have more developmental^ 
healthy high school experiences, to participate more in extracurricular activities, and to better 
avoid negative behaviors. 

The first three rows of Table 3.4 show the distribution of time spent on homework 
among Academy and non-Academy students in the high-risk subgroup. These estimates sug- 
gest that there were no systematic differences in the amount of time that these students spent 
on homework. 

Table 3.4 does indicate that the Academies increased the percentage of students who 
reported spending some time in extracurricular activities. Specifically, 69 percent of students 
in the high-risk non-Academy group indicated that they did not spend any time in extracur- 



9 NCES, 1995. 
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ricular activities during their 12 th -grade year. By contrast, 59 percent of the Academy students 
indicated that they had not participated in extracurricular activities. To the extent that partici- 
pation in extracurricular activities is an effective measure of engagement, this indicates that, 
among the high-risk subgroup, Academy students were more engaged in school than their non- 
Academy counterparts. 



Table 3.4 

Career Academies Evaluation 



Impacts on Experiences During the 12th Grade Year 
for Students in the High-Risk Subgroup 





Academy 


Non-Academy 


Percent 


Impact 


Outcome 


Group 


Group Impact 


Change 3 


per Enrollee 



Use of non-school hours 



Average time spent on homework (%) 



Less than 1 hour per week 


49.8 


52.5 


-2.7 


-5.2 


-3.5 


2 to 6 hours per week 


34.7 


36.6 


-1.9 


-5.3 


-2.3 


More than 6 hours per week 


15.5 


10.9 


4.6 


42.8 


5.6 


Average time spent on extra- 
curricular activities (%) 


None 


58.5 


68.9 


-10.4 ** 


-15.2 


-12.5 


1 to 4 hours per week 


23.5 


18.4 


5.2 


28.2 


6.2 


More than 4 hours per week 


18.0 


12.7 


5.3 


41.4 


6.3 


Youth development experiences 
Reported any positive youth 


development experiences in past year (%) 


62.7 


55.2 


7.5 


13.5 


9.0 


Worked on a volunteer project 


41.9 


31.6 


10.3 ** 


32.7 


12.4 


Received award for participation 


in athletics or a school organization 


40.6 


34.8 


5.9 


16.9 


7.0 


Received an academic award or scholarship 


26.6 


15.5 


11.1 ** 


71.6 


13.3 


Reported any risk-taking behaviors 


in past year (%) d 


34.8 


39.1 


-4.3 


-1 1.0 


- 5.2 


Has become a parent or is pregnant 


20.5 


21.4 


-1.0 


-4.5 


-1.1 


Has been expelled from school 


9.0 


8.3 


0.8 


9.4 


0.9 


Has come to school high on drugs or alcohol 


8.0 


11.1 


-3.1 


-28.3 


-3.8 


Has been arrested 


8.6 


13.4 


* 

oo 

■'T 

i 


-36.0 


-5.8 


Sample size (N=366) 


202 


164 









(continued) 
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Table 3.4 (continued) 

SOURCE: MDRC calculations from Career Academies Evaluation 12th Grade Survey Database. 

NOTES: 12 th grade year indicates the year that students were projected to reach the 12 th grade, when they initially 
enrolled in the Career Academy or regular high school program. Estimates are regression-adjusted using ordinary 
least squares, controlling for background characteristics of sample members. Rounding may cause slight 
discrepancies in calculating differences. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In both cases, 
statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Percent change is defined as the impact divided by the non-Academy group average. 

b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy and non- 
Academy group members ever enrolled in a Career Academy. It is italicized because its calculation does not 
involve a direct comparison of Academy and non-Academy students. 

“Students reported one or more of the positive youth development submeasures. 
d Students reported one or more of the risk-taking behaviors submeasures. 



The bottom panel of Table 3.4 lists the percentages of students who reported participation 
in various positive and negative youth development experiences. Specifically, the 12th Grade 
Survey asked students whether, during the past year, they had worked on a volunteer project in 
their community, received an award or recognition for participation in an athletic team or school 
organization, or received an academic award or scholarship. The 12th Grade Survey also asked 
students whether they had become a parent or were currently pregnant, had been expelled from 
school, had come to school high on drugs or alcohol, or had been arrested. 

The findings presented in Table 3.4 indicate that the Career Academies increased high- 
risk students’ involvement in volunteer projects and increased the likelihood that they received 
recognition for academic performance. Academies also reduced the percentage of students who 
had been arrested. 

E. Impacts on Plans and Steps Taken Toward Post-Secondary Education 
and Work 

The 12th Grade survey also asked students about their plans and preparation for college 
and work. Table 3.5 presents a summary of the impacts that Career Academies had on students’ 
future plans, the steps they took toward admission to a two- or four-year college, and their over- 
all outlook for the future. The findings for the high-risk subgroup indicate that Career Academies 
had a small impact on students’ educational plans for the next year, at least in terms of reducing 
the likelihood that they reported being unsure of whether they were going to school or work. The 
Academies also increased the percentage of students in the high-risk subgroup who took a num- 
ber of important steps toward attending a two-year or four-year college, and they enhanced stu- 
dents’ ultimate expectations for their educational attainment. 

The top four rows of the table present the distribution of students who planned to attend 
school, work, or combine the two during the year following their scheduled graduation. In gen- 
eral, the table shows that the vast majority of students in both the Academy and the non- 
Academy groups planned to combine school and work. The estimates indicate that approximately 
the same proportions of Academy and non-Academy students planned to attend school only, 
work only, or combine school and work. The last row in this section of the table does show that 
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Academy students were somewhat less likely than their non-Academy counterparts to report that 
they did not know whether they would attend school or go to work during the following year. 
Thus, the Academies appear to help students plan for some type of constructive activities after 
high school. 

The Academies increased the extent to which students in the high-risk subgroup were 
prepared to execute their plans for post-secondary education and employment. Table 3.5 shows 
the percentages of Academy and non-Academy students in the high-risk subgroup who reported 
completing a variety of steps needed to apply for and attend college or to find a job. These ac- 
tivities included collecting information about two- and four-year colleges, taking the SATs or 
ACTs, submitting an application, and having an interview. The measures in Table 3.5 indicate 
the percentage of students who reported completing these activities, 10 as well as the percentage of 
students who completed various activities aimed at securing a job during the following year. 

Table 3.5 first indicates that the Academies significantly increased the percentage of stu- 
dents in the high-risk subgroup who researched college options, took the SATs or ACTs, and 
submitted an application to a two-year or four-year college. For example, it shows that 22 per- 
cent of non-Academy students reported taking the SATs or ACTs during their 1 2 ,h -grade year, 
compared with 35 percent of Academy students in the high-risk subgroup. This 12 percentage 
point difference represents a 55 percent increase over the non-Academy group average. In addi- 
tion, just over half of the Academy group students reported that they had submitted an applica- 
tion to a two-year or a four-year college, compared with 35 percent of the students in the non- 
Academy group. 

Table 3.5 also shows that the Academies produced slight increases in the percentages of 
students in the high-risk subgroup who took concrete steps toward post-secondary employment, 
although none of these differences was statistically significant. 

Finally, the last several rows in Table 3.5 provide an indication of students’ educational 
expectations and general outlook for their future. Most notably, the Academy students were more 
likely than their non-Academy counterparts to report that they expected to graduate from college. 



IH. Career Academy Impacts for Students m the Low-Risk Subgroup 

The behavior and performance of the students in the low-risk non-Academy group pro- 
vide the best indication of how these students performed in high school without the opportunity 
to attend an Academy. In general, these students were unlikely to drop out and appeared to re- 
main engaged in high school on a number of dimensions. In all, only 3 percent of the low-risk 
non-Academy group dropped out of high school before the end of their 12 th -grade year, and ap- 
proximately three-quarters had earned sufficient credits to meet their districts’ graduation re- 
quirements. Just over 60 percent had completed the basic core academic curriculum, and over 
one-third had completed the more intensive core curriculum. On average, these students scored at 



'“Students were asked about these activities in terms of their efforts to attend a two-year or a four-year college. 
Information-gathering activities included talking with a teacher or other advisor about college, looking at college 
catalogues, visiting a college campus, and talking with one’s parents about how to pay for college. The measures 
presented in Table 3.5 indicate the percentage of students who reported engaging in two or more of these activities. 
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about the 40 th percentile nationally on a standardized math and reading achievement test. Sixty 
percent of non -Academy students in the low-risk subgroup reported that they had taken the SATs 
or ACTs, and nearly 80 percent reported that they had submitted an application to a two-year or a 
four-year college by the end of their 12 th -grade year. 



Table 3.5 

Career Academies Evaluation 



Impacts on Planning and Preparation for College and Work 
for Students in the High-Risk Subgroup 





Academy 


Non-Academy 




Percent 


Impact 


Outcome 


Group 


Group 


Impact 


Change 3 


per Enrollee 


Plans for next year (%) 












School only 


8.7 


6.7 


2.0 


30.4 


2.4 


Work only 


8.3 


7.1 


1.2 


17.2 


1.5 


Combine school and work 


79.3 


78.9 


0.4 


0.5 


0.4 


Unknown 


3.7 


7.3 


-3.6 ** 


-49.7 


-4.4 


Steps taken toward 2-year or 4-year 
college admission 












Researched college options 0 


82.1 


72.5 


9.6*** 


13.2 


11.5 


Took SATs or ACTs 


34.5 


22.3 


12.2 ** 


55.0 


14.7 


Submitted an application 


50.5 


35.3 


15.2*** 


43.1 


18.3 


Had an interview 


24.8 


18.5 


6.3 


34.1 


7.6 


Steps taken toward post-secondary employment (%) 












Talked with a teacher or advisor about a job 


44.1 


43.8 


0.3 


0.6 


0.3 


Submitted an application for employment 


60.7 


55.8 


4.9 


8.7 


5.8 


Interviewed for a position 


46.2 


43.6 


2.7 


6.2 


3.2 


Has previous work experience with prospective 
employer 


35.3 


31.1 


4.2 


13.5 


5.0 


Education expectations (%) 












Complete some post-secondary education 


91.6 


91.7 


-0.1 


-0.1 


- 0.1 


Attend college 


71.3 


66.1 


5.2 


7.9 


6.3 


Graduate from college 


53.7 


45.3 


8.3 * 


18.4 


10.0 


Has overall positive outlook for the future 0 


72.6 


66.0 


6.6 


10.0 


7.9 


Sample size (N=366) 


202 


164 








(continued) 
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Table 3.5 (continued) 

SOURCE: MDRC calculations from Career Academies Evaluation 12th Grade Survey Database. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for background characteristics 
of sample members. Rounding may cause slight discrepancies in calculating differences. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In both cases, 
statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
a Percent change is defined as the impact divided by the non-Academy group average. 

b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy and non- 
Academy group members ever enrolled in a Career Academy. It is italicized because its calculation does not involve 
a direct comparison of Academy and non-Academy students. 

indicates student engaged in two or more of the following activities: talking with teachers or advisors about 
college, discussing financing with parents, looking at college catalogues, or visiting schools. 

d Overall positive outlook for the future was defined by high ratings on questions about expectations for 
achievement, potential for attaining jobs, and knowledge of methods of finding jobs. 



For students in the low-risk Academy group, the findings discussed in this section indi- 
cate that, on most outcome measures, they remained engaged in high school at levels similar to 
their non-Academy counterparts. The findings also show, however, that the Career Academies 
did improve several important outcomes. Academies significantly increased the likelihood that 
these students would earn sufficient credits to meet their districts’ graduation requirements. They 
also increased career-related and vocational course-taking while enabling students in the low-risk 
subgroup to keep pace with their non-Academy peers in academic course-taking. The Academy 
students were somewhat less likely than their non-Academy counterparts to report that they had 
submitted an application to a two-year or a four-year college by the end of their 12 th -grade year. 
Finally, the Academies did not produce systematic changes in the low-risk Academy group’s 
math or reading achievement test scores. 

Table 3.6 provides a summary of Career Academy impacts on the school enrollment 
status and attendance rates of students in the low-risk subgroup. The first row in the table shows 
the percentages of Academy and non-Academy students who enrolled in a Career Academy. 
Most notably, it shows that 10 percent of students in the non-Academy group enrolled in a Ca- 
reer Academy. Although 8 percent of the non-Academy group students were identified as being 
enrolled in an Academy at the end of 12 th grade, very few of these students were enrolled in an 
Academy throughout high school. Further analysis of this finding indicated that most of the non- 
Academy students enrolled in an Academy in 11 th or 12 th grade, typically because they were in- 
terested in taking elective classes that were offered only within the Career Academies. 

The second row of Table 3.6 shows the percentage of students in the low-risk subgroup 
who remained enrolled in an Academy through the end of their 12 th -grade year. A relatively high 
percentage in the Academy group (nearly 74 percent of those initially selected) remained in the 
programs through the end of 12 th grade. This represents 82 percent of those who initially enrolled 
— a rate that is particularly high compared with the retention rate of students in the high-risk 
subgroup. 





Table 3.6 

Career Academies Evaluation 



Impacts on School Enrollment and Attendance 
for Students in the Low-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent Impact per 

Change 3 Enrollee b 


Ever enrolled in a Career Academy 
during high school (%) 


89.5 


10.2 


79 3 *** 




Was enrolled in a Career Academy 
at the end of grade 12 (%) 


73.5 


8.3 


65.2*** 




Dropped out of high school 
before the end of grade 1 2 (%) 


1.9 


2.9 


-1.0 


-34.4 -1.2 


Average attendance, 
grades 9-12 (%) 


95.0 


94.2 


0.8 


0.8 1.0 


More than 95 percent average 
attendance, grades 9-12 (%) 


67.3 


68.9 


-1.6 


-2.3 -2.0 


Less than 85 percent average 
attendance, grades 9-12 (%) 


2.2 


7.5 


-5.3 


-70.8 -6.7 


Sample size (N=385) 


204 


181 




, 



SOURCE: See Table 3.1. 



NOTES: See Table 3.1. 

A. Im pacts on School Enrollment and Attendance 

Table 3.6 reports the dropout rates and average attendance rates for students in the low- 
risk subgroup. This data indicate that Career Academies had no significant impact on these stu- 
dents’ attendance or retention in high school. A likely explanation for this finding is that very 
few students in the “low-risk” subgroup dropped out of high school, and few exhibited atten- 
dance problems or chronic absenteeism. 

In general, Table 3.6 shows that students in the low-risk subgroup were likely to remain 
enrolled in high school and to attend regularly, regardless of whether they were in a Career 
Academy. Very few students in the low-risk subgroup dropped out of high school (3 percent of 
the non-Academy group and 2 percent of the Academy group). Although this is in sharp contrast 
to the relatively high dropout rates among students in the high-risk subgroup, it was difficult for 
the Career Academies to improve on the very low dropout rates of non-Academy students in the 
low-risk subgroup. Similarly, average attendance rates were nearly 95 percent throughout high 
school for Academy and non-Academy group students. Interestingly, for the low-risk subgroup, 
the Academies did produce a slight reduction in chronic absenteeism (defined as having atten- 
dance rates of 85 percent or less throughout high school), although this difference was not statis- 
tically significant. 



O 
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B. Impacts on Credits Earned and Course-Taking 

Table 3.7 shows the distribution of course credits earned by Academy and non-Academy 
students in the low-risk subgroup. It indicates that Career Academies increased the percentage of 
these students who earned enough total credits to meet their districts’ graduation requirements. In 
particular, while approximately 75 percent of non-Academy students in this subgroup earned 
enough credits to meet the graduation requirement, nearly 86 percent of students in the Academy 
group did so. This 11 percentage point difference represents a 15 percent increase over the non- 
Academy group average. This finding suggests that although students in the low-risk subgroup of 
the study sample were highly unlikely to drop out of high school, a significant portion did not 
remain on course to earn enough credits to graduate on time. The Career Academies increased 
the likelihood that they would. 

The Career Academies also changed the mix of courses that students in the low-risk sub- 
group completed during high school. Most notably, the Academies substantially increased the 
number of non-academic courses these students completed, particularly career-related and voca- 
tional courses. For example, the last row of Table 3.7 indicates that 42 percent of the low-risk 
non-Academy group completed three or more credits in career-related or vocational courses, 
compared with approximately 77 percent of students in the low-risk Academy group. This is 
more than an 80 percent increase in career-related or vocational course-taking. 

Table 3.7 also shows that this increase did not come at the expense of students’ complet- 
ing academic courses, nor did it reduce the percentage of students in the low-risk subgroup who 
completed either the basic or the more intensive core academic curriculum. For example, 85 per- 
cent of the Academy group and approximately 89 percent of the non-Academy group completed 
12 or more academic credits (a difference that was not statistically significant). 

Table 3.7 does suggest, however, that the increase in career-related and vocational 
courses may have led the Academy group students to take fewer foreign-language courses. For 
example, 74 percent of the non-Academy group students earned at least two course credits in a 
foreign language, compared with 50 percent of the Academy group students. This represents 
nearly a one-third reduction in the non-Academy group average. It is not clear how this potential 
tradeoff may affect Academy students’ attractiveness to colleges, particularly four-year colleges 
that may prefer or require students to complete a foreign-language sequence during high school. 

C. Impacts on Math and Reading Achievement Test Scores 

Table 3.8 presents estimates of the impact of Career Academies on standardized measures 
of student achievement for the low-risk subgroup. In general, these findings do not reveal any 
systematic differences between Academy and non-Academy students; they exhibited similar lev- 
els of academic achievement as measured by standardized tests. 

Both Academy and non-Academy students in the low-risk subgroup had math scores av- 
eraging at about the 39 lh percentile nationally. The vast majority of students in both groups per- 
formed at the basic proficiency level or higher, and just over 40 percent of both groups scored at 
the middle proficiency level or higher. Although the reading test scores of Academy students in 
the low-risk subgroup were slightly lower than those of non-Academy students, the differences 
were not statistically significant. 
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Table 3.7 

Career Academies Evaluation 



Impacts on Credits Earned and Course-Taking 
for Students in the Low-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 0 


Impact per 
Enrollee h 


Credits earned 












Total course credits 


24.4 


23.6 


0.8 


3.2 


0.9 


Total course credits meet the 
graduation requirement (%) 


85.7 


74.6 


11.1 ** 


14.9 


14.0 


Earned 1 2 or more academic 
course credits (%) 


85.0 


88.5 


-3.5 


-3.9 


-4.4 


Earned 8 or more non-academic 
course credits (%) 


68.4 


51.1 


17.2 *** 


33.7 


21.7 


Course-taking 












English (4), Social Studies (3), 
Math (3), Science (3) c (%) 


39.2 


36.3 


2.9 


8.1 


3.7 


English (4), Social Studies (3), 
Math (2), Science (2) c (%) 


58.5 


61.2 


-2.7 


-4.4 


-3.4 


Earned 2 or more foreign-language 
credits (%) 


49.5 


73.6 


-24.1 *** 


-32.8 


- 30.4 


Earned 1/2 or more computer 
credits (%) 


59.2 


65.4 


-6.1 


-9.4 


-7.7 


Earned 3 or more career/vocational 
credits (%) 


76.5 


42.0 


34.6 *** 


82.4 


43.6 


Sample size (N=385) 


204 


181 









SOURCE: See Table 3.2. 



NOTES: See Table 3.2. 
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Table 3.8 

Career Academies Evaluation 



Impacts on Achievement Test Scores 
for Students in the Low-Risk Subgroup 

Academy Non-Academy Percent Impact per 

Outcome Group Group Impact Change 3 Enrollee b _ 



39.1 



-0.5 



- 1.2 



- 0.6 



Math achievement test scores 




Average national percentile 


38.6 


Proficiency level (%) 

1 : Rote memory operations 
3: Simple problem-solving 


91.0 

45.8 


Reading achievement test scores 




Average national percentile 


37.3 


Proficiency level (%) 

1 : Simple comprehension 
2: Simple inferences 


86.6 

40.0 


Sample size (N=147) 


80 



86.0 


5.0 


5.9 


6.3 


42.3 


3.5 


8.3 


4.4 



39.8 


-2.5 


-6.3 


-5.2 


95.6 


-9.0 


-9.4 


-11.4 


49.2 


-9.2 


-18.7 


-11.6 



67 



SOURCE: See Table 3.3. 
NOTES: See Table 3.3. 



B. Im pacts on Youth Development Experiences 

Table 3.9 shows the impacts that Career Academies had on a variety of student experi- 
ences during 12 th grade. In general, these estimates do not reveal substantial differences between 
Academy and non-Academy students in the low-risk subgroup, who exhibited similar levels of 
participation both in positive youth development activities and in negative risk-taking behaviors. 
Career Academies did produce a modest increase in the percentage of students in the low-risk 
subgroup who were involved in a volunteer project during their 12 th -grade year. 

E. Impacts on Plains and Steps Taken Toward Post-Secondary Education 
aud Work 

Table 3.10 lists a set of indicators of students’ plans and preparation for education and 
work during the year following their 12 ,h -grade year. It shows that the vast majority of Academy 
and non-Academy students in the low-risk subgroup reported that they planned both to work and 
to go to school in the following year. The second-to-last row of the table also indicates that about 
three-quarters of both Academy and non-Academy students reported that they eventually expect 
to graduate from college. 



O 
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Table 3.9 

Career Academies Evaluation 



Impacts on Experiences During the 12th Grade Year 
for Students in the Low-Risk Subgroup 





Academy 


Non-Academy 


Percent Impact 


Outcome 


Group 


Group Impact 


Change a per Enrollee 



Use of non-school hours 



Average time spent on homework (%) 



Less than 1 hour per week 


23.0 


24.6 


-1.6 


-6.6 


-2.0 


2 to 6 hours per week 


58.4 


49.6 


8.8 * 


17.7 


11.0 


More than 6 hours per week 


18.7 


25.8 


-7.1 * 


-27.6 


-9.0 


Average time spent on extra- 
curricular activities (%) 


None 


36.5 


33.0 


3.6 


10.8 


4.5 


1 to 4 hours per week 


29.0 


33.4 


-4.4 


-13.3 


-5.6 


More than 4 hours per week 


34.5 


33.6 


0.9 


2.6 


1.1 


Youth development experiences 
Reported any positive youth 


development experiences in past year (%) c 


80.3 


75.9 


4.4 


5.7 


5.5 


Worked on a volunteer project 


65.8 


50.0 


15.9*** 


31.7 


20.0 


Received award for participation 


in athletics or a school organization 


56.0 


56.2 


-0.2 


-0.4 


-0.3 


Received an academic award or scholarship 


41.7 


36.9 


4.8 


12.9 


6.0 


Reported any risk-taking behaviors 


in past year (%) 


15.6 


16.1 


-0.5 


-2.9 


-0.6 


Has become a parent or is pregnant 


6.0 


4.6 


1.4 


30.5 


1.8 


Has been expelled from school 


1.7 


2.0 


-0.3 


-13.3 


- 0.3 


Has come to school high on drugs or alcohol 


6.5 


9.7 


-3.3 


-33.6 


- 4.1 


Has been arrested 


4.4 


4.5 


-0.1 


-1.8 


- 0.1 


Sample size (N=389) 


218 


171 









SOURCE: See Table 3.4. 



NOTES: See Table 3.4. 
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Table 3.10 



Career Academies Evaluation 



Impacts on Planning and Preparation for College and Work 
for Students in the Low-Risk Subgroup 





Academy 


Non-Academy 




Percent 


Impact 


Outcome 


Group 


Group 


Impact 


Change 3 


per Enrollee 


Plans for next year (%) 












School only 


9.2 


8.9 


0.3 


2.9 


0.3 


Work only 


2.2 


2.6 


-0.4 


-15.7 


- 0.5 


Combine school and work 


86.3 


86.9 


-0.6 


-0.6 


-i 0.7 


Unknown 


2.4 


1.7 


0.7 


41.9 


0.9 


Steps taken toward 2-year or 4-year 
college admission 












Researched college options 0 


92.7 


96.3 


-3.7 


-3.8 


- 4.6 


Took SATs or ACTs 


60.0 


60.0 


-0.1 


-0.1 


- 0.1 


Submitted an application 


70.8 


78.7 


-7.9 * 


-10.0 


- 10.0 


Had an interview 


23.0 


29.8 


-6.7 


-22.6 


-5.5 


Steps taken toward post-secondary employment (%) 












Talked with a teacher or advisor about a job 


47.6 


37.8 


9.8 * 


25.9 


12.4 


Submitted an application for employment 


55.8 


56.6 


-0.8 


-1.5 


- 1.1 


Interviewed for a position 


38.7 


38.7 


-0.1 


-0.2 


-1 0.1 


Has previous work experience with prospective 
employer 


27.7 


26.8 


0.9 


3.4 


1.2 


Education expectations (%) 












Complete some post-secondary education 


97.4 


97.3 


0.0 


0.0 


0.0 


Attend college 


88.7 


85.8 


2.8 


3.3 


3.6 


Graduate from college 


75.2 


74.5 


0.6 


0.9 


0.8 


Has overall positive outlook for the future 0 


78.4 


78.1 


0.2 


0.3 


0.3 


Sample size (N=389) 


218 


171 








SOURCE: See Table 3.5. 



NOTES: See Table 3.5. 



Table 3.10 does indicate that students in the Academy group were somewhat less likely 
than their non- Academy counterparts to have submitted an application to a two-year or four-year 
college by the end of their 12 lh -grade year. Interestingly, over 90 percent of students in both 
groups had investigated college options, and 60 percent had taken the SATs or ACTs — a critical 
step in applying for college. However, whereas 79 percent of students in the non- Academy group 
had submitted an application to college, only 71 percent of Academy students in the low-risk 
subgroup had done so. 
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The reason for this lower rate of applying for college is not clear. One hypothesis is that 
Academy students in the low-risk subgroup may have been opting to go to work rather than to 
college. In general, there does not appear to be any support for this hypothesis. First, as noted 
earlier, Academy and non-Academy students were about equally likely to report that they 
planned to work or to combine work and school in the year following graduation. Table 3.10 also 
indicates that the groups were about equally likely to have applied or interviewed for a job for 
the following year. Another hypothesis is that Academy students may have been more likely to 
plan on attending a two-year college, many of which may not require a formal application to be 
submitted while students are still in high school. The 12th Grade Survey did not ask students to 
differentiate between activities aimed at a two-year as opposed to a four-year college. 

The findings regarding students’ post-secondary activities are not clear. A longer follow- 
up period is needed to determine the actual college enrollment and completion rates of the Acad- 
emy and non-Academy groups and to assess the types of college programs they attend. This is a 
key feature of the second phase of the Career Academies Evaluation, which includes asking stu- 
dents in the study sample to complete a survey 12 months after their scheduled graduation from 
high school and again four years after their scheduled graduation. 



IV. Career Academy Impacts for Stun demits |m the Medium-Risk Subgroup 

The final subgroup of students include those who were not at particularly high risk of 
dropping out of high school, but appeared to exhibit at least a moderate level of disengagement 
from school. This medium-risk subgroup represents about half the student sample. Overall, 8 
percent of non-Academy students in the medium-risk subgroup dropped out of high school be- 
fore the end of their 12 th -grade year, and approximately two-thirds earned sufficient credits to 
meet their districts’ graduation requirements. Just over half completed the basic core academic 
curriculum, and about 30 percent completed the more intensive core curriculum. On average, 
these students scored at about the 30 th percentile nationally on a standardized math and achieve- 
ment test and at the 35 th percentile in a standardized reading test. About 47 percent of non- 
Academy students in the medium-risk subgroup reported that they had taken the SATs or ACTs, 
and over 60 percent reported that they had submitted an application to a two-year or a four-year 
college by the end of their 12 th -grade year. 

The findings discussed in this section indicate that, on average, the Career Academies had 
little or no impact on most outcomes for students in the medium-risk subgroup. In other words, 
most outcome levels for students in the Academy group were about the same as they were for 
students in the non-Academy group. As discussed in Chapter 5, however, these overall averages 
for students in the medium-risk subgroup mask a high degree of variation in impacts across the 
sites in the study. 

Table 3.11 provides a summary of Career Academy impacts on the school enrollment 
status and attendance rates of students in the medium-risk subgroup. The first two rows of the 
table show that 89 percent of the Academy group enrolled in an Academy during at least one 
semester of high school and that 59 percent remained enrolled in the programs through the end 
of their 12 th -grade year. This means that, on average, about two-thirds of students in the me- 
dium-risk subgroup who initially enrolled in an Academy remained in the program throughout 
high school. 
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A. Im pacts on School Enrollment and Attendance 



Table 3.11 shows the average school enrollment and attendance outcomes among stu- 
dents in the medium-risk subgroup. The data suggest that the Career Academies had no impact 
on dropout prevention or average attendance for this subgroup. The third row of the table pro- 
vides an estimate of the dropout rate among Academy and non-Academy students in this sub- 
group. It shows that 9 percent of Academy students in the medium-risk subgroup dropped out of 
high school, compared with 8 percent of their non-Academy peers. This difference was not sta- 
tistically significant. Similarly, the next three rows present estimates of several different meas- 
ures of attendance among Academy and non-Academy students in this subgroup. The results 
suggest no systematic differences in the attendance patterns of Academy and non-Academy stu- 
dents in the medium-risk subgroup. 

Table 3.11 

Career Academies Evaluation 



Impacts on School Enrollment and Attendance 
for Students in the Medium-Risk Subgroup 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 8 


Impact per 
enrollee b 


Ever enrolled in a Career Academy 
during high school (%) 


89.0 


6.7 


82.3 *** 






Was enrolled in a Career Academy 
at the end of grade 1 2 (%) 


58.8 


3.0 


55.9*** 






Dropped out of high school 
before the end of grade 1 2 (%) 


9.0 


8.0 


1.0 


12.1 


1.2 


Average attendance, 
grades 9-12 (%) 


88.4 


89.6 


-1.2 


-1.4 


-1.5 


More than 95 percent average 
attendance, grades 9-12 (%) 


36.2 


37.7 


-1.5 


-3.9 


-1.8 


Less than 85 percent average 
attendance, grades 9-12 (%) 


22.4 


19.3 


3.1 


15.9 


3.7 


Sample size (N=724) 


393 


331 









SOURCE: See Table 3.1. 



NOTES: See Table 3.1. 
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B. Im pacts on Credits Earned and Course-Taking 

Table 3.12 presents measures of course-taking patterns for Academy and non- Academy 
students in the medium-risk subgroup. The top panel of the table indicates that the Academies 
did not have a systematic effect on the number of credits students earned in high school or on the 
number of credits they earned in academic and non-academic courses. Also, about 65 percent of 
students in both the medium-risk Academy and non-Academy groups had earned sufficient cred- 
its to meet their districts’ graduation requirements. 

Table 3.12 

Career Academies Evaluation 



Impacts on Credits Earned and Course-Taking 
for Students in the Medium-Risk Subgroup 



Outcome 


Academy 

Group 


Non- Academy 
Group 


Impact 


Percent 

Change 3 


Impact per 
Enrollee b 


Credits earned 












Total course credits 


22.6 


22.9 


-0.4 


-1.6 


- 0.4 


Total course credits meet the 
graduation requirement (%) 


64.8 


65.2 


-0.3 


-0.5 


-0.4 


Earned 1 2 or more academic 
course credits (%) 


69.8 


. 69.4 


0.4 


0.6 


0.5 


Earned 8 or more non- academic 
course credits (%) 


59.1 


56.9 


2.2 


3.9 


2.7 


Course-taking 












English (4), Social Studies (3), 
Math (3), Science (3) c (%) 


28.8 


30.5 


-1.7 


-5.6 


- 2.1 


English (4), Social Studies (3), 
Math (2), Science (2 ) c (%) 


48.7 


51.0 


-2.3 


-4.4 


- 2.7 


Earned 2 or more foreign- language 
credits (%) 


42.2 


49.0 


i 

ON 

oo 

* 


-13.9 


-8.3 


Earned 1/2 or more computer 
credits (%) 


66.3 


57.7 


8.6 ** 


15.0 


10.5 


Earned 3 or more career/vocational 
credits (%) 


65.7 


48.4 


yj ^ *** 


35.9 


21.1 


Sample size (N=724) 


393 


331 









SOURCE: See Table 3.2. 



NOTES: See Table 3.2. 
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Table 3.12 does indicate that the Academies produced changes in some specific subject 
areas for the medium-risk subgroups. Academy students were more likely to complete three or 
more credits in career-related or vocational courses and were more likely to complete at least 
one semester of computer-related courses. At the same time, the Academies reduced the per- 
centage of students who completed a sequence of at least two years in a foreign language. 
Given that there was no overall change in the total number of non-academic credits earned by 
students in the medium-risk subgroups, it appears that the Academy students were taking the 
career-related and computer-related courses instead of other non-Academy courses or elec- 
tives. Also, because there was no overall reduction in academic course-taking, it appears that 
Academy students were more likely than their non-Academy counterparts to take other aca- 
demic courses instead of a foreign language. 

C. Impacts on Math and Reading Achievement Test Scores 

Table 3.13 summarizes the results for students in the medium-risk subgroup who took the 
NELS: 88 reading and math achievement tests. It appears that the Academies produced a slight 
reduction in reading test scores. On average, students in the non-Academy group scored at the 
35 01 percentile nationally, compared with the 31 st percentile for Academy group students. It is 
possible that some of this difference may be due to differences in the composition of the Acad- 
emy and non-Academy students who completed the achievement tests. This may have resulted 
from the fact that the medium-risk Academy students were somewhat more likely than medium- 
risk non-Academy students to complete the test. 

Table 3.13 

Career Academies Evaluation 



Impacts on Achievement Test Scores 
for Students in the Medium-Risk Subgroup 



Outcome 


Academy 

Group 


Non- Academy 
Group 


Impact 


Percent Impact per 

Change 3 Enrollee b 


Math achievement test scores 












Average national percentile 


29.1 


29.7 


-0.6 


-2.0 


-0.7 


Proficiency level (%) 












1 : Rote memory operations 


84.8 


80.9 


3.9 


4.8 


4.7 


3: Simple problem-solving 


28.5 


31.3 


-2.8 


-8.9 


-3.4 


Reading achievement test scores 












Average national percentile 


30.8 


35.1 


-4.3 * 


-12.2 


-5.2 


Proficiency level (%) 












1 : Simple comprehension 


80.1 


85.9 


-5.7 


-6.7 


-7.0 


2: Simple inferences 


39.7 


43.5 


-3.8 


-8.7 


-4.6 


Sample size (N=233) 


124 


109 








SOURCE: See Table 3.3. 



NOTES: See Table 3.3. 
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D. Impacts on Youth Development Experiences 

Table 3.14 lists measures of various high school experiences and extracurricular activities 
among Academy and non-Academy students in the medium-risk subgroup. In general, the table 
indicates that there was no systematic difference between Academy and non-Academy students 
in terms of their use of non-school hours or in their exposure to various positive or risk-related 
activities. 



Table 3.14 

Career Academies Evaluation 



Impacts on Experiences During the 12th Grade Year 
for Students in the Medium-Risk Subgroup 





Academy 


Non-Academy 


Percent Impact 


Outcome 


Group 


Group Impact 


Change 3 per Enrollee b 



Use of non-school hours 



Average time spent on homework (%) 



Less than 1 hour per week 


37.0 


32.9 


4.1 


12.4 


5.0 


2 to 6 hours per week 


47.9 


46.0 


1.9 


4.1 


2.3 


More than 6 hours per week 


15.1 


21.1 


-6.0 ** 


-28.3 


-7.2 


Average time spent on extra- 
curricular activities (%) 


None 


47.0 


52.3 


-5.3 


-10.1 


- 6.4 


1 to 4 hours per week 


30.3 


25.9 


4.4 


17.0 


5.3 


More than 4 hours per week 


22.7 


21.8 


0.9 


4.2 


1.1 


Youth development experiences 
Reported any positive youth 


development experiences in past year (%) c 


71.3 


69.2 


2.0 


2.9 


2.5 


Worked on a volunteer project 


53.7 


48.9 


4.8 


9.8 


5.8 


Received award for participation 


in athletics or a school organization 


46.7 


41.7 


5.0 


11.9 


6.0 


Received an academic award or scholarship 


28.0 


25.1 


2.8 


11.3 


3.4 


Reported any risk-taking behaviors 


in past year (%) 


23.8 


25.6 


-1.8 


-7.2 


-2.2 


Has become a parent or is pregnant 


11.1 


10.8 


0.4 


3.6 


0.5 


Has been expelled from school 


3.9 


6.0 


-2.1 


-35.5 


- 2.6 


Has come to school high on drugs or alcohol 


8.4 


8.7 


-0.3 


-3.6 


- 0.4 


Has been arrested 


5.3 


5.2 


0.1 


1.8 


0.1 


Sample size (N=755) 


407 


348 









SOURCE: See Table 3.4. 



NOTES: See Table 3.4. 
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E. Im pacts on Plans and Steps Taken Toward Post-Secondarv Education 
and Work 

Table 3.15 presents findings on the Career Academies’ impacts on the medium-risk sub- 
group’s plans and preparation for post-secondary education and work. As with many outcomes 
discussed in this section, the table indicates that in the medium-risk subgroup there was no sys- 
tematic difference between Academy and non-Academy students in terms of their plans and 
preparation for post-secondary education and work. 

Table 3.15 

Career Academies Evaluation 



Impacts on Planning and Preparation for College and Work 
for Students in the Medium-Risk Subgroup 





Academy 


Non-Academy 




Percent 


Impact 


Outcome 


Group 


Group 


Impact 


Change 3 


per Enrollee b 


Plans for next year (%) 












School only 


8.3 


8.9 


-0.7 


-7.4 


-0.8 


Work only 


4.8 


5.1 


-0.3 


-5.5 


-0.3 


Combine school and work 


83.9 


83.9 


0.1 


0.1 


0.1 


Unknown 


3.0 


2.1 


0.9 


41.8 


1.1 


Steps taken toward 2-year or 4-year 
college admission 












Researched college options 0 


87.1 


85.7 


1.4 


1.6 


1.7 


Took SATs or ACTs 


44.3 


46.9 


-2.5 


-5.4 


-3.1 


Submitted an application 


62.6 


63.1 


-0.5 


-0.7 


-0.6 


Had an interview 


24.3 


25.6 


-1.3 


-5.0 


-1.6 


Steps taken toward post-secondary employment (%) 












Talked with a teacher or advisor about a job 


39.1 


40.9 


-1.8 


-4.4 


-2.2 


Submitted an application for employment 


60.7 


55.4 


5.3 


9.5 


6.4 


Interviewed for a position 


42.1 


41.9 


0.2 


0.6 


0.3 


Has previous work experience with prospective 
employer 


30.8 


26.8 


4.0 


14.9 


4.9 


Education expectations (%) 












Complete some post-secondary education 


93.6 


93.9 


-0.3 


-0.3 


- 0.3 


Attend college 


81.4 


81.2 


0.2 


0.3 


0.3 


Graduate from college 


63.4 


66.2 


-2.8 


-4.2 


-3.4 


Has overall positive outlook for the future d 


75.1 


78.5 


-3.4 


-4.4 


- 4.2 


Sample size (N=755) 


407 


348 








SOURCE: See Table 3.5. 



NOTES: See Table 3.5. 
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V. Career Academy Impacts Averaged Across the Student Subgroups 

As noted earlier, most previous studies of Career Academies have focused on findings 
that are averaged across the diverse groups of students they serve. To provide a sense of how the 
findings for these Career Academies might compare with other such averages, this section sum- 
marizes impact findings for the full study sample. A key conclusion from this analysis is that 
these overall averages mask a great deal of variation in the potential that Academies have to 
make a difference for students, particularly for students at risk of school failure. 

Table 3.16 lists key outcomes that were discussed in previous sections of the chapter. In 
general, the pattern of impacts is consistent with the pattern seen in the subgroups, although the 

Table 3.16 

Career Academies Evaluation 



Impacts on Selected High School Outcomes for 
Students in the Study Sample 



Outcome 


Academy 

Group 


Non-Academy 

Group 


Impact 


Percent 

Change 3 


Impact per 
Enrollee b 


Ever enrolled in a Career Academy 
during high school (%) 


88.5 


6.8 


81.7 *** 






Dropped out of high school 
before the end of grade 12 (%) 


10.1 


12.4 


-2.4 


-19.0 


-2.9 


Total course credits meet the 
graduation requirement (%) 


64.8 


58.8 


6.0 ** 


10.2 


7.3 


English (4), Social Studies (3), 
Math (2), Science (2) c (%) 


47.8 


46.3 


1.4 


3.1 


1.8 


English (4), Social Studies (3), 
Math (3), Science (3) c (%) 


28.4 


26.7 


1.7 


6.3 


2.1 


Reading achievement test score 
average national percentile d 


31.2 


32.9 


-1.7 


-5.3 


-2.1 


Math achievement test score 
average national percentile d 


29.9 


29.4 


0.5 


1.7 


0.6 


Reported any positive youth development 
experiences in past year (%) e 


71.6 


67.4 


4.2 * 


6.2 


5.1 


Reported any risk-taking behaviors 
in past year (%) f 


24.2 


26.6 


-2.4 


-8.9 


-2.9 


Submitted application for 2- or 4-year college 


62.0 


60.2 


1.8 


3.0 


2.2 


Submitted application for post-secondary 
employment (%) 


59.4 


55.8 


3.6 


6.5 


4.4 


Sample size (N= ) 



(continued) 
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Table 3.16 (continued) 



SOURCES: MDRC calculations from Career Academies Evaluation Student School Records, 12th Grade 
Achievement Test, and 12th Grade Survey Databases. 

NOTES: Credits include zero values for grades in which sample members were identified as school dropouts. 
Estimates are regression-adjusted using ordinary least squares, controlling for background characteristics of sample 
members. Rounding may cause slight discrepancies in calculating differences. All measures indicate credits earned up 
until the end of the 1 2 th -grade year. 1 2 th -grade year indicates the year that students were projected to reach the 1 2 th - 
grade when they initially enrolled in the Career Academy or regular high school program. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In both cases, 
statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
a Percent change is defined as the impact divided by the non-Academy group average. 

b Impact per enrollee is defined as the impact divided by the difference in the percentage of Academy and non- 
Academy group members ever enrolled in a Career Academy. It is italicized because its calculation does not involve a 
direct comparison of Academy and non-Academy students. 

C N umbers refer to the amount of credits that were earned in each subject area. 

d The reading and math achievement tests are the cognitive battery tests of reading and mathematics used in the 
NELS: 88 study. There were a total of five proficiency levels for mathematics and three for reading. Particular 
proficiency levels are reported in the table to illustrate general trends in performance in the distribution of students. 
Percentile scores reflect students' performance in relation to a nationally representative sample of H^-graders. 
c Students reported one or more of the positive youth development submeasures. 
f Students reported one or more of the risk-taking behaviors submeasures. 



magnitude of impacts for the full study sample is smaller. On average, across all the participating 
students and sites, the Academies produced increases in credits earned toward graduation, par- 
ticularly in career-related and vocational courses. They also increased students’ exposure to 
positive youth development activities. Not surprisingly, the averaged impacts tend to look more 
like those for the medium-risk subgroup, the largest of the three. 



VI. Conclusions 

The evidence presented in this chapter suggests that the Career Academies have the 
strongest positive effects on the outcomes of students who begin high school at a high risk of 
school failure. The Academies reduce dropout rates, increase attendance, and increase credits 
earned in both academic and vocational subjects. They also appear to improve outcomes 
among students who are at low risk of school failure. Although these students are already 
highly engaged in school and are unlikely to drop out, the Academies appear to improve sev- 
eral outcomes, including the percentage of these students who earn enough credits to graduate 
on time. On the other hand, the Academies do not appear to have much effect on students in 
the medium-risk subgroup. 

Importantly, although the estimates in this chapter are focused on individual subgroups, 
they are aggregated across all the sites in the study. To the extent that differences in the imple- 
mentation of the Academy model affect the impact of the Career Academies, these estimates may 
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still mask important variation in the effects of Academies on students’ performance, engagement, 
and achievement in high school. To pursue this issue, Chapter 4 explores the variation across 
sites in the implementation of the Academy model, and Chapter 5 explores the effect of this 
variation on the impact of Academies on student outcomes. 
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Chapter 4 



Factors Associated with Stmdemt Outcomes amd the 
Pattern off Career Academy Impacts 



This chapter and Chapter 5 explore some potential pathways through which the core ele- 
ments of the Career Academy approach may change students’ educational and work-related out- 
comes. The findings from these chapters provide some further context for explaining and interpret- 
ing the pattern of impacts discussed in Chapter 3. 

Section I of this chapter identifies several aspects of students’ high school environments and 
experiences that are most strongly associated with positive outcomes that students may attain by the 
end of 12 th grade. In particular, it examines students’ perspectives on three school-related domains: 
the degree of interpersonal support they received from teachers and peers, the extent to which their 
classes included applied teaching and learning activities, and their level of exposure to career 
awareness and work-based learning activities. This analysis focuses on both Academy and non- 
Academy students to shed light on the relationship between these domains and students’ level of 
school engagement through the end of 1 2 th grade. 

Section II of this chapter examines differences across the participating sites in the extent to 
which the Career Academies increased the level of interpersonal support, applied learning, and 
work-related learning activities available to students. This analysis shows that a subset of 
Academies represent a large contrast with their non-Academy environments, particularly in terms 
of the interpersonal supports they offer to students. Chapter 5 explores the extent to which these 
Academies produced a different pattern of impacts on student outcomes than Academies that 
represented less of a contrast from their non-Academy environments. 



I. Potential Pathways to Positive Student Outcomes 

As discussed in Chapter 1 , the Career Academy approach has attracted a great deal of atten- 
tion in recent years, in part because its core features offer direct responses to a variety of problems 
that have been identified in high schools. Figure 1.1 provided a conceptual model showing the po- 
tential pathways through which these features may affect student outcomes in ways that are consis- 
tent with the broad range of goals that have been ascribed to Career Academies. Analyses con- 
ducted earlier in the Career Academies Evaluation, and updated for this report, provide empirical 
support for the conceptual framework illustrated in Figure 1.1. This section of the chapter briefly 
reviews the general findings from these analyses. 1 

Recall that the conceptual model listed four groups of constructs leading from the core 
elements of the Academy approach to various short- and long-term outcomes. The four sets of 
constructs are: 



'See Kemple 1997a, 1997b; and Kemple, Poglinco, and Snipes, 1999. 
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© Career Academy organizational elements that distinguish the Academy ap- 
proach from the regular high school environment in which it is implemented; 

© supports and learning opportunities that are intended to evolve from the or- 
ganizational elements; 

o high school outcomes that the Academies aim to improve by enhancing the 
supports and learning opportunities; and 

© post-secondary outcomes that reflect some of the long-term goals of the 
Academy approach. 

Analyses conducted for previous reports from this evaluation and updated for the current 
report have focused on testing the correlation between measures of constructs indicated in the sec- 
ond column of Figure 1.1 and measures of the outcomes indicated in the third column. In other 
words, the analyses have explored empirical relationships between supports and learning opportu- 
nities and the outcomes students attain during high school. 

One strand of analysis has examined the relationships between supports and outcomes for 
all students in the study sample, regardless of whether they had access to an Academy. This has 
been aimed at addressing such questions as whether students who experience higher levels of inter- 
personal support from teachers and peers are more likely to remain enrolled and engaged in school 
than students who experience lower levels of such supports. In fact, survey data collected for the 
evaluation indicate that students who reported receiving a high degree of support from their teach- 
ers and peers during 9 th and 1 0 th grades were less likely to drop out of high school and more likely 
to complete a core curriculum. 2 Similarly, students who participated intensively in career awareness 
and work-based learning activities tended to be more engaged in school and were more likely to be 
prepared to graduate and go on to college than those who did not participate in such activities or 
participated less intensively. 3 Finally, the analyses indicated a positive, yet weaker, association 
between students’ exposure to integrated and applied learning activities and their school engage- 
ment and performance. In general, however, the strongest associations have been found between the 
interpersonal supports (such as teachers’ expectations and peer collaboration) and various measures 
of student performance and engagement in school. 4 

To the extent that such relationships exist, on average, for all students in the study sample, it 
is likely that these types of supports serve as key pathways through which the Career Academies 
improve student engagement in school. As noted earlier, this is because the supports and learning 
opportunities are closely aligned with the organizational features of the Academy approach. For 
example, enhanced interpersonal support is likely to evolve from the intensive interaction and col- 



2 These supports included students’ perceptions of the personalized attention they receive from teachers, teachers’ 
expectations for student performance and achievement, engagement levels of classmates, and opportunities to work 
closely with classmates. 

3 This finding should be interpreted with some caution because participation in these activities typically occurred 
after 1 1 th grade. Some Career Academies, as well as other high school programs, tended to select students for their 
intensive career awareness and work-based learning activities based on students’ level of engagement and perform- 
ance in school. As a result, a high degree of school engagement may lead to higher levels of participation in career 
awareness and work-based learning activities, rather than the other way around. 

4 For additional information on these analyses, please contact the authors. 
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laboration offered by the school-within-a-school. A more focused curriculum and enriched teaching 
and learning are intended to develop through the Academy’s integration of academic and occupa- 
tional content. Greater exposure to career awareness and work-based education is promoted through 
the employer partnerships. 

In fact, as discussed in the previous reports from this evaluation, Academy students were 
considerably more likely to experience the types of support and learning opportunities listed in Fig- 
ure 1.1 than were their non- Academy counterparts in the study. It should be noted, however, that 
Career Academy impacts on supports and learning opportunities may not directly cause any of the 
impacts on such outcomes as dropout rates and credits earned toward graduation. Although these 
linkages make sense from an conceptual standpoint, it may be that the students who experienced 
the greatest increase in supports are different from the students who experienced the largest reduc- 
tions in dropout rates or the greatest increases in credits earned. This suggests that the Career Acad- 
emy impacts may follow from pathways other than those leading through the types of supports and 
learning opportunities listed in Figure 1.1. 

One way to further test these relationships is to identify subgroups of sites in the study 
where the Career Academies generated particularly large increases in the supports and learning op- 
portunities listed in Figure 1.1. The next step would be to determine whether the Academies in 
these sites also generated larger impacts on such outcomes as dropout rates and progress toward 
graduation. The next section of the chapter summarizes analyses that identify a group of sites that 
represented a particularly dramatic contrast with their non-Academy environments, at least in terms 
of the supports and learning opportunities discussed above. 



II. Sources of Variatiom Aroomig tflue Sites That Mav Be Associated 
with Differences in Impacts 

The primary focus of this report is on the difference between Academy and non-Academy 
environments and on the effects that this difference may have on students’ experiences and behav- 
iors. As discussed in previous reports and earlier in this report, all the Career Academies had im- 
plemented the core features of the Academy approach and represented a clear contrast with the non- 
Academy environments in their schools. This section of the report begins an exploration of whether 
some versions or contexts for the Academy approach produce larger impacts on student outcomes 
than others. 

As noted above and in previous reports from this study, the sites participating in the 
Career Academies Evaluation differ from one another along a number of dimensions. 5 As a 
result, there are numerous characteristics or criteria that might be used to differentiate the sites. 
For the purposes of this report, the strategy for exploring cross-site variation in the impacts of 
Career Academies focuses on similarities and differences among sites in terms of the contrast 
between their Academy and non-Academy environments, as opposed to exploring variation 
only on the basis of differences among the Career Academy programs. For example, some Ca- 
reer Academies with highly supportive school-within-a-school environments are located in 
high schools where the vast majority of students feel safe, challenged, and supported by their 



5 See Kemple and Rock, 1 996, for more detailed information about the similarities and differences among sites. 



teachers and peers. In such a context, the Academies may not add much to the high degree of 
support' already offered by the non-Academy environment. 

Conversely, other Academies are located in high schools where very high percentages of 
students do not receive the support or instruction required to keep them engaged in school and on 
a path toward graduation. Even though these Academies may not be as “well implemented” as 
Academies in other contexts, they provide much more support and challenge for their students 
than is available in the non-Academy environments. 

The strategy used in this report to highlight contrasts among the participating sites was 
guided by the theory of change described in Chapter 1 and by analyses of the relationships 
among the key constructs described above. As noted earlier, interpersonal supports were found to 
have the strongest relationship with later outcomes such as dropout prevention and progress toward 
graduation. In other words, students who reported a high degree of support from their teachers 
and peers in 9 th or 10 th grade were more likely, by the end of 12 th grade, to remain enrolled in 
high school, to have high attendance rates (95 percent or higher), and to have completed a core 
curriculum. In general, Career Academy students were more likely to experience high levels of 
interpersonal supports than their counterparts in non-Academy environments. 

Given the strong association between interpersonal supports and later outcomes, the 
primary construct used to distinguish among sites in the study was the difference in the level of 
interpersonal supports that Academy and non-Academy students received. In short, individual 
sites were ranked according to the difference between the percentage of Academy and non- 
Academy students who reported receiving a high level of support from teachers and peers. 
Such ranking indicated that a subset of five participating sites showed a particularly large dif- 
ference in the interpersonal supports of Academy and non-Academy students. Moreover, as a 
group, these sites generated larger differences in the school experiences of Academy and non- 
Academy students along several other important dimensions. Therefore, throughout the report, 
the Career Academies in these sites are referred to as high-contrast Academies 6 The remaining 
sites showed a substantially smaller difference in the level of teacher and peer support reported 
by Academy and non-Academy students, as well as somewhat smaller differences along other 
dimensions of the high school experience. The Career Academies in these sites are referred to 
as low-contrast Academies. 

Table 4.1 lists a variety of measures that highlight some of the key differences between 
high- and low-contrast sites and their Career Academies. 7 The first row in Table 4.1 shows the 
average percentages of Academy and non-Academy students who gave a high rating on an 
overall measure of teacher support. The left panel of the table shows the differences in these 
percentages for the high-contrast Academies, and the right panel shows the differences for the 
low-contrast Academies. The difference in this outcome was much larger for the high-contrast 

6 It is important to note that high-contrast Academies are not necessarily the most highly supportive or best- 
implemented Academies in the study. Rather, they are the sites where the Career Academies presented the greatest 
contrast with their non-Academy school environments according to the student survey data collected for the evalua- 
tion. In some low-contrast sites, a high percentage of Academy students reported receiving high levels of support 
from teachers and peers, but they are characterized as “low-contrast” sites because equally high or higher percent- 
ages of non-Academy students also reported receiving high levels of support. 

7 For a detailed description of the measures presented in Table 4.1, see Kemple, 1997a, 1997b; and Kemple, 
Poglinco, and Snipes, 1999. 
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SOURCES: MDRC calculations from Career Academies Evaluation Student School Experience Questionnaire Database and 12th Grade Survey Database. 
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Academies, where there was a 20 percentage point difference between Academy and non- 
Academy students. This represents an increase of 42 percent in the proportion of Academy stu- 
dents who felt that they received a high degree of support from their teachers. For the low- 
contrast Academies, there was no statistically significant difference between the percentages of 
Academy and non-Academy students who indicated that they received a high degree of support 
from their teachers. 

The first row of Table 4.1 further indicates that the contrast between the two groups of 
sites can be attributed mostly to differences between the Academy groups rather than to differ- 
ences between the non-Academy groups. In the high-contrast sites, 68 percent of the Academy 
students reported receiving a high level of teacher support, compared with 51 percent of the 
Academy students in the low-contrast sites. On the other hand, very similar percentages of the 
non-Academy students from both groups of sites reported that they received a high level of 
teacher support. This pattern suggests that a key difference between these two groups of sites re- 
flects differences among Academy implementation strategies or differences among Academy 
teachers (or, more likely, a combination of the two). There is little contrast — at least on this 
measure — in the non-Academy environments. 

The second row of Table 4.1 indicates a somewhat different pattern with respect to peer 
support. Again, there is a clear contrast in the difference in peer support reported by Academy 
and non-Academy students across the two groups of sites. In this case, however, the difference 
across the two groups of sites is driven both by differences between Academy students and by 
differences between non-Academy students. Academy students in the high-contrast sites were 
more likely to give a high rating on peer support than were Academy students in the low-contrast 
sites. At the same time, non-Academy students in the high-contrast sites were less likely to give a 
high rating on peer support than were non-Academy students in the low-contrast sites. 

What did the high-contrast Academies do that was different from the low-contrast Acade- 
mies? In general, they tended to implement tightly organized school-within-a-school organizations. 
Programs in high-contrast Academies tended to include a core group of four or five teachers whose 
responsibilities fell exclusively within the Academy. The vast majority of students in these pro- 
grams had their core courses scheduled in blocks within the Academy, and very few non-Academy 
students were included (except, for example, to ensure adequate enrollments). 8 These programs also 
tended to be located in a clearly identifiable area of the school building or campus. In addition, the 
Academy teachers in high-contrast sites tended to indicate that they had more opportunities to col- 
laborate with colleagues, that they felt part of a strong learning community, that they were able to 
influence key areas of their work, and that they emphasized personalized attention to their students. 9 

It is important to note, however, that considerable variation existed among the Academies 
within the two groups of sites. In some sites, for example, the high contrast between the Acade- 
mies and regular school environments appears to have stemmed from an advanced level of pro- 
gram implementation — a level that was closer to the ideal Academy model. In other cases, 
however, the high contrast appears to reflect that the regular high school environment was par- 
ticularly stressful and unsupportive, and so the Academies provided considerably higher levels of 

8 See Kemple, 1997a, 1997b. 

9 This information comes from site visits, teacher interviews, and classroom observations completed as part of 
the Career Academies Evaluation. 
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support even though their programs were less well implemented than programs in other sites. In 
some high-contrast sites, a relatively low percentage of Academy students reported high levels of 
teacher and peer support, but an even lower percentage of non-Academy students did so. 

; There was also variation among the low-contrast sites. In some of them, relatively large 
percentages of both Academy and non-Academy students reported high levels of support from 
teachers and peers; in others, relatively small percentages did so. This suggests that, in some 
cases, the smaller effects on interpersonal support in the low-contrast sites were partly a func- 
tion of weaker implementation of the Academy model. In other cases, however, the lack of ef- 
fect was a function of the Academies’ being implemented in environments that were already 
relatively supportive. 

The remainder of Table 4.1 reveals that the two groups of sites differed along several other 
dimensions of the Academy experience, including students’ exposure to enriched learning opportu- 
nities and participation in career awareness and work-based learning activities. In general, there 
were larger differences between the experiences of Academy and non-Academy students in the 
high-contrast sites than in the low-contrast sites. As discussed above and in Chapter 1, however, 
these constructs were not as strongly related to student engagement and performance. In addition, 
the variation within the two groups of sites was greater on these dimensions than on the interper- 
sonal supports. 

The analyses in Chapter 5 focus primarily on the differences in impacts between these 
two groups of sites. In general, the chapter explores the hypothesis that the high-contrast 
Academies produced larger impacts on student engagement and performance than the low- 
contrast Academies. 
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Chapter 5 



The Relationship Between Career Academy Implementation 

and Impacts 

This chapter explores the relationship between variation in Career Academy implementa- 
tion across the sites in the study and variation in the impacts the different Academies had on stu- 
dent outcomes. In particular, this chapter explores the extent to which the pattern of impacts dis- 
cussed in Chapter 3 differed across the high- and low-contrast sites identified in Chapter 4. 
Chapter 3 found that the positive effects of Career Academies were concentrated among students 
at a high risk of dropping out but that impacts were relatively modest among students who were 
not at such risk. The findings in Chapter 4 revealed that outcomes were strongly related to meas- 
ures of the interpersonal support that students experienced in their school environments; the 
Academies in this evaluation substantially varied in the magnitude of contrast between the level 
of interpersonal support experienced by Academy students and the level experienced by non- 
Academy students in the regular high school environment. 

In short, this chapter explores the hypothesis that Academies which represented the most 
dramatic contrast with their non-Academy environments produced larger and more consistent 
positive impacts than did Academies that were more similar to their non-Academy environments. 
Following is a summary of key findings discussed in this chapter. 

• Among students in the medium-risk subgroup, Career Academies that repre- 
sented the greatest contrast with the regular high school environment produced 
positive impacts, including lower dropout rates and increased completion of a 
core academic curriculum. 

• Academies that represented less contrast with the non-Academy environment 
in terms of interpersonal support produced some negative effects for students 
in the medium-risk subgroup, including higher dropout rates, reduced atten- 
dance, and lower rates of academic course-taking. 

• In general, both high-contrast and low-contrast Academies produced similar 
patterns of impacts among students in the high-risk subgroup. The primary 
difference across these sites is that the low-contrast Academies somewhat re- 
duced this subgroup’s dropout rates and produced much larger increases in 
vocational and career-related course-taking. 

• In general, both high-contrast and low-contrast Academies also produced 
similar patterns of impacts among students in the low-risk subgroup. The pri- 
mary difference across the sites for this subgroup is that the low-contrast 
Academies somewhat increased career-related and vocational credits while re- 
ducing academic course-taking. 

In sum, these findings do not clearly support the hypothesis that the high-contrast 
Academies produced larger and more consistently positive impacts than the low-contrast 
Academies. Although this pattern can be seen among students in the medium-risk subgroup, for 
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the high- and low-risk subgroups the impacts across the groups of sites had more similarities than 
differences. Nevertheless, the differences that do exist suggest two implications: 

o Increasing the level of interpersonal support for students — in addition to of- 
fering more opportunities to participate in career awareness and work-based 
learning activities — can produce a consistent pattern of positive effects for 
both the high- and the medium-risk subgroups (representing approximately 75 
percent of the students Academies serve). Under such circumstances, students 
in the low-risk Academy group are likely to do at least as well as their non- 
Academy counterparts. 

® If Academies do not increase the level of interpersonal support (again, relative 
to the regular school environment), they may actually reduce engagement 
among the medium-risk subgroup, and they may also lead the low-risk sub- 
group to replace some academic courses with career-related or vocational 
courses. 

These findings and implications are discussed in greater detail below. 



I. Contrasting Impacts for Students in the Medium-Risk Subgroup 

Tables 5.1 and 5.2 report the impacts among students in the medium-risk subgroup (that 
is, students who were not identified as being at a particularly high or low risk of dropping out). 
The left panel of each table presents findings for the high-contrast Academies, and the right panel 
presents findings for the low-contrast Academies. 1 

The findings in the left panel of Table 5.1 indicate that, among students in the medium- 
risk subgroup, the high-contrast Academies had a positive impact on a number of key outcomes. 
In particular, the high-contrast Academies appeared to reduce dropout rates and increase the per- 
centage of students who completed a core academic curriculum. Specifically, the high-contrast 
Academies reduced dropout rates for the medium-risk subgroup from 11 percent among non- 
Academy students to 5 percent among their Academy group counterparts. This 6 percentage 
point difference represents a 54 percent reduction in the dropout rate for non-Academy students 
in the medium-risk subgroup. In addition, the high-contrast Academies increased the percentage 
of the medium-risk subgroup who completed a basic core academic curriculum. Whereas 49 per- 
cent of non-Academy students in these sites completed the curriculum, 58 percent of Academy 
students did so. 

The right panel of Table 5.1 indicates that impacts at the low-contrast Academies oc- 
curred in the opposite direction from impacts at the high-contrast Academies. In particular, the 



'In addition to providing statistical significance tests of the impacts within each group of sites, the tables in this 
chapter provide statistical significance tests of the difference in impacts across the two groups of sites. The dagger 
symbols (f) in the rightmost column of each table indicate whether or not the impact for the high-contrast Acade- 
mies was statistically significantly different from the impact observed for the low-contrast Academies. If a differ- 
ence in impacts is statistically significant, one may have greater confidence that it is a systematic difference between 
the groups of sites and does not arise merely from chance. 
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low-contrast Academies appear to have increased dropout rates for the medium-risk subgroup, 
from 5 percent among the non-Academy students to 13 percent among the Academy students. 
The dropout rate for Academy students in these sites was about two and a halftimes larger than it 
was for their non-Academy counterparts. Furthermore, while neither group of Academies pro- 
duced statistically significant changes in the percentage of students in the medium-risk subgroup 
who earned enough credits to graduate on time, the patterns of impacts run in opposite directions. 
That is, the high-contrast Academies increased the percentage of students who earned enough 
credits to meet the districts’ graduation requirements, while the low-contrast Academies de- 
creased this percentage. (Note that the difference in these impacts is statistically significant even 
though the individual impact estimates are not.) 

There were particularly dramatic differences in impacts on course-taking patterns be- 
tween the high-contrast and low-contrast Academies. The low-contrast Academies reduced the 
percentage of students in the medium-risk subgroup who completed the basic core academic cur- 
riculum, from 54 percent for students in the non-Academy group to 37 percent for students in the 
Academy group. A similar reduction occurred in the percentage of students who completed two 
or more foreign-language courses (from 50 percent for the non-Academy group to 26 percent for 
the Academy group). 

The last row of Table 5.1 indicates that the low-contrast Academies produced more than 
twice as large an impact on vocational course-taking than did the high-contrast Academies. Specifi- 
cally, in the high-contrast Academies, the percentage of students in the medium-risk subgroups who 
completed three or more career-related or vocational courses increased from 50 percent for the non- 
Academy group to 62 percent for the Academy group. This 12 percentage point difference repre- 
sents an increase of about 23 percent over the non-Academy group average. In the low-contrast 
Academies, however, the percentage of students who completed three or more career-related or vo- 
cational courses increased from 46 percent for the non-Academy group to 72 percent for the Acad- 
emy group — an increase of 57 percent over the non-Academy group average. 

Table 5.2 presents the impacts of each group of Academies on the medium-risk sub- 
group’s youth development activities and the steps these students took toward post-secondary 
education or employment opportunities. The primary differences between the two groups of sites 
relate to the steps students took toward employment. The high-contrast Academies slightly re- 
duced the percentages both of students who applied for post-secondary employment and of stu- 
dents who interviewed for a position, whereas the low-contrast Academies had significant posi- 
tive impacts on both outcomes. 

Although the two groups of sites were differentiated by the level of interpersonal support 
that the Academies offered to students (relative to the non-Academy environment), it is notable 
that several differences in impacts relate to curriculum and work-related activities. The patterns 
of findings presented in Tables 5.1 and 5.2 suggest that the low-contrast Academies placed 
greater emphasis on career-related and vocational course-taking and on helping students take 
concrete steps toward post-secondary employment opportunities. Nonetheless, the high-contrast 
Academies do appear to have produced stronger impacts on school engagement for students in 
the medium-risk subgroup. 
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II. Contrasting Impacts for Students in the High-Risk Subgroup 

Tables 5.3 and 5.4 present the impact findings for the high-risk subgroup. These findings 
indicate that-the patterns of impacts across the high-contrast and low-contrast Academies are 
generally similar. Both groups of Academies had positive effects on dropout rates, attendance, 
credits earned toward graduation, and completion of a core academic curriculum. With the ex- 
ception of the impacts on career-related and vocational course-taking, none of the differences in 
impacts across the site groups is statistically significant. 2 

There are two notable exceptions to the pattern of similar impacts for students in the 
high-risk subgroups across the sites. First, although both groups of Academies produced reduc- 
tions in dropout rates, the impact on dropout rates for the high-contrast Academies was not sta- 
tistically significant. (Note that the differences in dropout rate impacts across the groups of sites 
is not statistically significant.) In particular, the low-contrast Academies cut dropout rates in half 
for the high-risk subgroups, while the high-contrast Academies reduced dropout rates by about 
25 percent. There is no clear explanation for this pattern of findings, and it runs counter to the 
hypothesis that increased interpersonal support in the high-contrast sites should have produced 
larger reductions in dropout rates. 

Despite the somewhat smaller impact on dropout rate reduction, however, Table 5.3 shows 
that the high-contrast Academies produced comparable impacts on attendance and credits. For ex- 
ample, the high-contrast Academies increased the percentage of the high-risk subgroup who earned 
sufficient credits to graduate, from 30 percent for the non-Academy students to 43 percent for the 
Academy students. A similar pattern occurred among the low-contrast Academies. Whereas 19 per- 
cent of the high-risk non-Academy group at the low-contrast sites earned enough credits to graduate 
by the end of 12 th grade, 36 percent of their peers in the Academy group did so — a difference of 87 
percent. Again, although this impact appears to be somewhat larger among the low-contrast 
Academies, the difference in impacts is not statistically significant. 

Both high-contrast and low-contrast Academies increased the percentage of the high-risk 
subgroup who completed the basic core academic curriculum. Again, while this impact is statis- 
tically significant only in the high-contrast sites, there is no significant difference across the sites 
in the magnitude of this impact. In fact, to the extent that there is a difference, the percentage 
change at the low-contrast sites appears to be larger than at the high-contrast sites. 

As noted above, vocational or career-related course-taking reflects the only dimension 
along which the difference in impacts across sites is statistically significant. While the impact 
on the percentage of students who took three or more vocational courses is significant for both 
high-contrast and low-contrast Academies, the impact on vocational course-taking is substan- 
tially larger for low-contrast Academies. At these sites, 28 percent of the high-risk non- 
Academy group completed at least three career-related or vocational courses, compared with 
66 percent of the Academy group. This impact represents an increase of 141 percent over the 
non-Academy group and is nearly four times larger than the impact for the high-contrast 
Academies. This suggests that while the high-contrast and low-contrast Academies produced 



2 In other words, one cannot reject the hypothesis that the impacts for high-contrast Academies are the same as 
the impacts for low-contrast Academies. 
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similar patterns of impacts on student engagement generally, the low-contrast Academies pro- 
duced a much larger impact on career-related and vocational course-taking. It is important to 
note that, for the high-risk subgroup, this increase in career-related and vocational course-taking 
did not appear to result in a reduction in academic course-taking. 

Table 5.4 presents the impacts for the high-contrast and low-contrast Academies on youth 
development outcomes and steps taken toward post-secondary education and employment. In 
general, the data suggest no real differences in impacts between the two groups of sites. 

III. Contrasting Impacts for Students in the Low-Risk Subgroup 

Table 5.5 presents the impact findings for the low-risk subgroups across the high-contrast 
and low-contrast Academies. Not surprisingly, the findings suggest that neither group of Acade- 
mies had a meaningful impact on dropout rates or attendance. Students in the low-risk subgroup 
at both groups of sites were unlikely to drop out of high school and had relatively high atten- 
dance rates, even in the absence of the Academy treatment. Given these outcome levels, there 
was little opportunity for either group of Academies to make much difference. 

Table 5.5 does show that both groups of Academies increased the percentage of the low- 
risk subgroup who earned enough credits to meet their districts’ graduation requirements. The 
table indicates that, at the low-contrast sites, 75 percent of the non-Academy students earned 
enough credits by the end of their scheduled 12 th -grade year, compared with 88 percent of the 
Academy students. Similarly, at the high-contrast sites, 73 percent of the non-Academy students 
earned enough credits to graduate on time, compared with 84 percent of the Academy students in 
the low-risk subgroup. 

The last three rows of Table 5.5 show that the primary differences in impacts across the 
two groups of sites occurred in course-taking patterns. The low-contrast Academies reduced the 
percentage of students in the low-risk subgroup who completed the basic academic curriculum as 
well as the percentage who completed two or more foreign-language courses. On the other hand, 
the high-contrast Academies produced a slight increase in the percentage of students who com- 
pleted the basic core curriculum and a smaller (not statistically significant) reduction in the per- 
centage taking two or more foreign-language courses. 

Table 5.5 indicates that, at the low-contrast sites, 64 percent of the non-Academy students 
in the low-risk subgroup completed the basic core academic curriculum. In comparison, 54 per- 
cent of Academy students did so. Although this difference is not statistically significant, it is sig- 
nificantly different from the pattern at the high-contrast sites, where the Academies increased by 
almost the same magnitude the percentage of students who completed the basic core curriculum. 

Table 5.5 also indicates that the low-contrast Academies reduced the percentage of the 
low-risk subgroup who completed two or more foreign-language courses. While 79 percent of 
the non-Academy students at these sites earned two or more foreign-language credits, 44 percent 
of their Academy counterparts did so. This difference is statistically significant and represents a 
reduction of 44 percent. Although there is a negative effect on this outcome for the high-contrast 
sites, it is not statistically significant; moreover, it is substantially smaller than the effect at the 
low-contrast sites. 
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Finally, while both groups of Academies increased the vocational course-taking for stu- 
dents in the low-risk subgroup, the impact at low-contrast sites was slightly larger. The high- 
contrast Academies increased the percentage of students who earned three or more career-related 
or vocational credits from 44 to 73 percent (an increase of 65 percent), whereas the low-contrast 
Academies increased this percentage from 41 to 80 percent (an increase of 94 percent). 

Table 5.6 presents findings on the low-risk subgroup’s youth development experiences 
and steps taken toward post-secondary education and employment. In general, there were few 
impacts on these outcomes and few differences in the patterns at high-contrast and low-contrast 
Academies. The one exception is the effect of Academies on positive youth development experi- 
ences. The first row of Table 5.6 presents the percentages of the low-risk subgroup who reported 
such positive youth development experiences as working on a volunteer project in their commu- 
nity, receiving an award or recognition for participating in an athletic team or school organiza- 
tion, and receiving an academic award or scholarship. Among the high-contrast Academies, 70 
percent of the low-risk non-Academy group reported positive youth development experiences, 
compared with 84 percent of the Academy students. The low-contrast Academies did not have an 
impact on this outcome. 



The findings reported in this chapter have several implications for policy and practice. 
First, it appears that Academies which produced the largest increases in interpersonal supports 
for students (relative to their non-Academy environments) also produced positive impacts on 
student engagement among both the high-risk and the medium-risk subgroups (which represent 
about 75 percent of the students they serve). Thus, focusing on interpersonal supports appears to 
be a particularly important factor for both policymakers and practitioners when implementing 
Academies. As discussed in Chapter 4, such supports as increased teachers’ expectations, per- 
sonalized attention, and students’ connections with an engaged peer group are most likely to be 
derived from the school-within-a-school component of the Academy approach. This component 
may be a necessary, though perhaps not sufficient, condition to keep students in school and to 
provide a foundation for improving their achievement. 

Second, it appears that high-contrast and low-contrast Academies may differ in important 
ways other than in their levels of interpersonal support. This is suggested by the larger impacts 
that the low-contrast Academies had on career-related and vocational course-taking and by their 
reductions in academic course-taking, particularly among the medium-risk and low-risk sub- 
groups. Given that completion of a core academic curriculum and foreign-language courses are 
often key requirements for admission to a four-year college, to the extent that the mission of the 
Academies has evolved to include preparation for work and college, it is important that the 
Academies avoid limiting opportunities for any subset of students. Therefore, policymakers and 
practitioners may need to ensure that Academies are implemented in a way that increases both 
interpersonal support and exposure to career-related themes and experiences in school but that 
does not limit students’ opportunities to complete key academic courses. 
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As discussed in Chapter 1 , the primary data used in this report were obtained from three 
sources: school transcript records, a survey administered to students at the end of their 1 2 lh -grade 
year, and a standardized math and reading achievement test administered to a subsample of stu- 
dents at the end of their H^-grade year. This appendix presents the percentage of students in the 
full study sample for whom these data are available. It also examines the comparability of stu- 
dents in the Academy and non-Academy groups for whom data are availabile. 



I. Data Availability for Students in the Study Samp le 

As noted in Chapter 1, MDRC attempted to obtain data for a sample of 1,764 students 
from nine of the sites selected for the study. 1 For the purposes of this report, this group of stu- 
dents is referred to as the study sample. Of the students in the study sample, 959 (55 percent) 
were randomly selected to enroll in an Academy. For the purposes of this report, these students 
are referred to as the Academy group. The remaining 805 students (45 percent of the study sam- 
ple) were not invited to participate in the Academies but could choose other options available in 
their high school or school district. These students constitute the study’s control group and are 
referred to in this report as the non-Academy group. 

These groups of students were identified over a three-year period including the 1992- 
93, 1993-94, and 1994-95 school years. The students in the study sample were identified at the 
end of their 8 th - or 9 m -grade year, depending on when they applied for an Academy program. 
Recall that two of the Academies began in the 9 th grade and that the remaining seven began in 
the 10 th grade. Students applied for admission to the programs at the end of the year prior to 
expected enrollment. This report follows students in the study sample through the end of the 
year they were scheduled to graduate from high school. This corresponds to the 1995-96, 
1996-97, or 1997-98 school year, depending on the year and grade level when a student en- 
tered the study. In short, MDRC attempted to collect data for students over the four-year pe- 
riod they were scheduled to enroll in high school. 

Table A.l lists the percentages of students in the Academy and non-Academy groups for 
whom each of the key data sources is available. These percentages are referred to here as response 
rates. The top panel of the table shows the response rates for the full study sample, and the bottom 
three panels show the response rates for each of the three risk subgroups discussed throughout the 
report. Although not shown in the table, the overall response rates are as follows: Student School 
Records data are available for just over 82 percent of the students in the study sample, and 12th 
Grade Survey data are available for just over 85 percent of the study sample. These response rates 



'As discussed in Kemple and Rock, 1996, the initial research sample consisted of 1,953 students from 10 sites. 
A total of 1 89 of these students were dropped from the initial research sample, and efforts to collect data for them 
were discontinued. These students include the following. First, as noted in Chapter 1, one of the initial Career 
Academies was disbanded after two years in the study and was unable to provide sufficient follow-up data for its 
students in the study sample. Thus, the 126 students in the research sample from this site are not included in the 
study sample for this report. Second, MDRC learned that 59 of the students in the initial research sample applied for 
an Academy program during their 10 th -grade year and should not have been included in the study. This information 
was obtained from pre-random assignment school records and was confirmed with school staff. Finally, over the 
course of the data collection period, MDRC learned through contact with the schools and families that four addi- 
tional students were deceased. 
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Table A.l 

Career Academies Evaluation 

Data Availability for the Full Study Sample and the Risk Subgroups 



Subgroup and Data Source 


Academy 

Group 


Non-Academy 

Group 


Difference 


Full studv sample 








In Student School Records Database (%) 


81.5 


83.5 


-2.0 


In 12th Grade Survey Database (%) 


86.2 


84.8 


1.4 


In 12th Grade Achievement Test Database 3 (%) 


71.8 


69.9 


1.9 


Sample size (n= 1,764) 


959 


805 




High-risk subgroup 








In Student School Records Database (%) 


71.7 


74.1 


-2.4 


In 12th Grade Survey Database (%) 


78.3 


75.9 


2.4 


In 12th Grade Achievement Test Database 3 (%) 


60.0 


56.0 


4.0 


Sample size (n=474) 


258 


216 




Medium-risk subgroup 








In Student School Records Database (%) 


83.4 


83.2 


0.2 


In 12th Grade Survey Database (%) 


86.4 


87.4 


-1.0 


In 12th Grade Achievement Test Database 3 (%) 


74.3 


69.9 


4.4 


Sample size (n=869) 


471 


398 




Low-risk subgroup 








In Student School Records Database (%) 


88.7 


94.8 


-6.1 * 


In 12th Grade Survey Database (%) 


94.8 


89.5 


5.3 * 


In 12th Grade Achievement Test Database 3 (%) 


80.0 


84.8 


-4.8 • 


Sample size (n=421) 


230 


191 





SOURCES: MDRC calculations from Career Academies Evaluation Student Baseline Questionnaire 
Database, Student School Records Database, and 12th Grade Survey Database. 



NOTES: The statistical significance of the difference between Academy and non-Academy groups is 
indicated as *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Percentages based on those attempted for the 12th-grade achievement test: 

Academy Non-Academy 

Group Group 



Full sample 


372 


319 


High-risk subgroup 


105 


84 


Medium-risk subgroup 


167 


156 


Low-risk subgroup 


100 


79 



1/ BEST copy available 
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are considered to be high, particularly given that they cover a four-year follow-up period. Typically, 
program evaluations such as this aim for response rates of 80 percent or higher. The 12th Grade 
Achievement Test data are available for approximately 71 percent of the students in the study sam- 
ple who were attempted for the test administration. 2 

The top panel of Table A.l shows that there are no systematic differences between the 
Academy and non-Academy groups in the proportion of students for whom these data are available. 
Thus, although these data are not available for all students in the study sample, data availability is 
virtually the same for the Academy and non-Academy groups. The second and third panels in the 
table show that there also are no systematic differences in response rates for each of the data 
sources for the high-risk and medium-risk subgroups. 

There are slight differences in response rates, however, between Academy and non- 
Academy students in the low-risk subgroup. In particular, response rates for Student School Rec- 
ords data are somewhat higher among the students in the low-risk non-Academy group than they 
are for the students in the low-risk Academy group. Conversely, response rates for the 12th Grade 
Survey are somewhat higher among students in the low-risk Academy group. When response rates 
are larger for one research group, impact estimates may be biased slightly if there are systematic 
differences in background characteristics and pre-random assignment experiences between Acad- 
emy and non-Academy students who did respond. As discussed in the next section of this Appen- 
dix, there are no systematic differences between Academy and non-Academy students in any of the 
subgroups for any of the data sources. 

A key question for interpreting the findings presented in this report is whether students for 
whom data are available are representative of the full study sample. To address this question, mul- 
tiple regression was used to determine the extent to which the average characteristics of the students 
with data differed from the average characteristics of students for whom data are not available. This 
analysis was carried out for each of the three data sources. In each case, the analysis indicated that 
there were systematic differences in background characteristics between students with data and 
those without data. An illustration of the differences can be seen by comparing the response rates of 
the high-risk, medium-risk, and low-risk subgroups in Table A.l. Across all three data sources, re- 
sponse rates are lowest for the students in the high-risk subgroup and are highest for students in the 
low-risk subgroup. 

In short, the analysis of response rates indicates that the samples of students for whom data 
are available are not completely representative of the full study sample of 1,764 students. Thus, 
caution should be exercised when attempting to generalize the findings beyond the students who 
are included in the analyses. Nevertheless, the overall response rates show that data are available for 
the vast majority of students in the study sample, making the findings reflective of the behavior of 
most of the sample. 



2 

As noted in Chapter 1, MDRC attempted to administer the achievement test to the 691 students in the study 
sample who were scheduled to be in the 12 th grade at the end of the 1997-98 school year. 



-i oo- 



ERIC 





II. Comparison of Characteristics of Academy and Non-Academv Groups 
in the Database Samples 

The unique strength of a random assignment research design is that it yields two groups 
for which there are no systematic differences in measured and unmeasured background charac- 
teristics at the time sample members are identified for the study. Because the two groups entered 
the study with equivalent characteristics, any differences that emerge after that point can be at- 
tributed with confidence to the fact that one group had access to an Academy and the other group 
did not. 



Table 2.1 in Chapter 2 presents, one at a time, average characteristics of Academy and 
non-Academy students in the full study sample. This table shows that there are not statistically 
significant differences between the groups on any of the characteristics. A more rigorous way to 
test for such differences is to use multiple regression analysis. Table A.2 presents linear regres- 
sion estimates and statistical tests of whether there are any systematic differences between Acad- 
emy and non-Academy students in the full study sample and in each of the three risk subgroups. 
The first column in Table A.2 shows that only one characteristic (age at application to Academy) 
is statistically significant and that there is no systematic difference. The final entry in the column, 
the p-value of the F-statistic, is very close to 1, providing strong evidence that there is no overall 
pattern of differences between Academy and non-Academy students in the full study sample. A 
p-value of .1 or lower is typically used to indicate a “high” likelihood that there are systematic 
differences between the groups. 

The three remaining columns in Table A.2 present the same analysis for each of the three 
risk subgroups. These columns indicate that there are slight differences in a few individual char- 
acteristics but no overall pattern of differences between Academy and non-Academy students for 
any of the subgroups. The p-value of the F-statistic for each subgroup ranges from .767 to .879. 

As discussed in the previous section of this appendix, MDR.C obtained school records for 
approximately 82 percent of the full sample; obtained 12th Grade Survey data for approximately 
84 percent of the full study sample; and obtained 12th Grade Achievement Test data for ap- 
proximately 71 percent of those attempted for the test. Thus, the Student School Records Data- 
base sample consists of 1,454 students; the 12th Grade Survey Database sample consists of 1,510 
students; and the 12th Grade Achievement Test Database sample consists of 490 students. 

A key question underlying the analyses presented in this report is: Do these response 
patterns preserve the random assignment design? In other words, does each of the database sam- 
ples exhibit the same lack of systematic differences between Academy and non-Academy stu- 
dents, both overall and for each of the risk subgroups? To assess this question, regression analy- 
ses were used in the same manner as exhibited in Table A.2. Table A.3 presents the results for 
the Student School Records Database sample; Table A.4 presents the results for the 12th Grade 
Survey Database sample; and Table A.5 presents the results for the 12th Grade Achievement Test 
Database sample. 

These tables each indicate slight differences in a few particular characteristics, but there are 
no systematic differences between the Academy and non-Academy groups for any of the database 
samples. This is true for the full study sample and for each of the risk subgroups. Given the overall 
lack of differences in background characteristics between the two groups, one can be confident 
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that differences in the outcome measures used throughout the report were caused by one group’s 
having had access to the Career Academies and the other group’s not having had such access. 

In summary, random assignment created two groups of students without systematic over- 
all differences in background characteristics and prior school experiences. The pattern of re- 
sponse rates for each of the data sources preserves this feature of the research design. The lack of 
systematic differences between the Academy and non- Academy research groups is also pre- 
served within each of the risk subgroups that are used throughout the report. 
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Table A.2 (continued) 

Full Study Sample High-Risk Subgroup Medium-Risk Subgroup Low-Risk Subgroup 

Parameter Parameter Parameter Parameter 

Estimate Estimate Estimate Estimate 

Variable . (Standard Error) (Standard Error) (Standard Error) (Standard Error) 
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Table A.4 (continued) 

Full Study Sample High-Risk Subgroup Medium-Risk Subgroup Low-Risk Subgroup 
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(0.061) (0.118) (0.100) (0.427) 

Is overage for grade level 0.026 0.005 0.039 2.770 

, (0.079) (0.154) (0.123) (2.509) 

(continued) 



Full Study Sample High-Risk Subgroup Medium-Risk Subgroup Low-Risk Subgroup 

Parameter Parameter Parameter Parameter 

Estimate Estimate Estimate Estimate 

Variable (Standard Error) (Standard Error) (Standard Error) (Standard Error) 
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Appendix B 



Strategies for Creating Subgroups of Students 
Defined by Characteristics Associated with Risk 
of Dropping Out 



Much of the analysis presented in this report focuses on subgroups of students defined by 
background characteristics and prior school experiences associated with dropping out of high 
school. This appendix explains the manner in which these subgroups were created, including the 
rationale behind this strategy and the implications it has for interpreting the findings presented in 
this report. 



I. Analytic Importance of Subgroup Analysis 

A central theme that has emerged from the Career Academies Evaluation thus far is that 
in order to understand the impact of the programs, it is important to recognize the heterogeneity 
of the student population and the likelihood that some groups of students may benefit differently 
than others. As discussed in Chapter 3, when the impact results are averaged across the diverse 
groups of students the Career Academies served, it appears that the programs produced only 
slight reductions in dropout rates and modest improvements in students’ progress toward 
graduation and increases in participation in youth development activities. These aggregate results 
mask the high degree of variation in the Career Academies’ potential to make a difference and in 
the actual differences they made for some students. In short, findings that are aggregated across 
the diverse groups of students served by the Academies are unlikely to reveal many of the most 
important effects that Academies have. Positive effects for some subgroups of students may be 
offset or muted by small or zero impacts for other subgroups. 

For example, an important goal of the Career Academies is to reduce dropout rates and 
increase students’ engagement in school. As noted earlier in the evaluation, Career Academies 
serve a broad cross section of students, many of whom enter the programs highly engaged in 
school. It is unlikely that the programs will have an effect on dropout rates among these students, 
who are highly unlikely to drop out of school even if they do not attend an Academy. On the 
other hand, a number of students in the sample who applied for the Academies were relatively 
disengaged from high school and appeared to be at risk of dropping out of high school. To the 
extent that the Academies can have an effect on dropout rates, it is likely to be concentrated 
among these students. The magnitude of this effect could be diluted or even completely hidden if 
averaged with the lack of impact for the rest of the students in the sample. 

In order to assess the effect of the Academies more sensitively, therefore, it was neces- 
sary to differentiate among students with different needs and trajectories at the time they entered 
the Academy. The attempt to make distinctions among groups of individuals with different needs 
and characteristics, who might experience substantially different benefits from an intervention, is 
not uncommon to experimental research in general or to education research in particular. An im- 
portant goal of these subgroup strategies is often to make distinctions among groups of individu- 
als who, in the absence of the treatment under study, would have experienced substantially dif- 
ferent outcomes. 

The random assignment research design used in this evaluation provides a unique oppor- 
tunity to identify subgroups of students who, without access to an Academy, were relatively 
highly likely to drop out of high school and to compare them with similar students who did have 
access to an Academy. The use of the random assignment research design is relatively rare in the 
context of large-scale evaluations of education programs, particularly at the secondary school 
level. Not only does such a design provide the unusual opportunity to establish which outcomes 
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would have been observed in the absence of the Academy treatment, but it also provides an op- 
portunity to observe the relationships between background characteristics and important out- 
comes in the absence of the intervention. 

There are several strategies for identifying subgroups. The following section describes a 
more traditional approach and highlights several limitations that led to the use of a strategy that 
provides greater insight into the variation in program effects. 



II. Traditional Approach to Defining Subgroups: Risk-Factor 
Accumulation 

One of the strategies most frequently used to define subgroups might be called “risk- 
factor accumulation.” It entails first identifying a list of background characteristics typically as- 
sociated with an important outcome or with the manner in which the program treatment is likely 
to be delivered. A critical outcome for many high school interventions, including the Career 
Academy’s approach, is dropping out of high school. A number of education research studies 
have identified several background characteristics and prior school experiences that are associ- 
ated with a high likelihood of dropping out of high school. This includes prior school experi- 
ences such as poor attendance, low grades, or being held back in a previous grade. It also in- 
cludes demographic characteristics such as being from a low-income family, having a sibling 
who dropped out, or having moved and transferred schools several times. 

The risk-factor accumulation strategy classifies students into risk subgroups by counting 
the number of risk factors an individual has, weighting all the factors equally. For example, if 
one identified six characteristics associated with dropping out, individuals with two or more of 
these characteristics might be considered to be at “high risk” of dropping out; those with only 
one of the characteristics might be considered to be at “moderate” or “medium” risk of dropping 
out; and those with none of the characteristic factors might be considered at “low risk.” 

This strategy has the appeal of being straightforward in execution, and it can be translated 
directly into a strategy for targeting students to receive special services. For example, if a par- 
ticular school intervention were found to prevent students in the high-risk subgroup from drop- 
ping out, teachers or administrators might wish to ensure that students with two or more of the 
risk characteristics be included in that program. 

At the same time, the accumulation strategy has several important limitations. First, such 
an analysis gives equal weight to each of the risk-related background characteristics and prior 
school experiences examined. As a result, it does not account for the fact that some characteristics 
are more highly associated with school failure than others. This strategy also does not account for 
the fact that some characteristics are associated with school success and may offset the risk associ- 
ated with other characteristics. As a result, it fails to account for the possibility that, given the same 
number of risk factors, different combinations of characteristics may indicate different degrees of 
risk. In other words, because some characteristics are more strongly associated with academic out- 
comes than others, students with the same number of characteristics may actually be substantially 
more or less likely than one another to drop out of high school. Finally, this strategy is based on 
categorical variables and is therefore unable to take advantage of the more subtle distinctions 
among students that are captured by continuous variables. 
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Because it does not allow for a more complex set of relationships between risk factors 
and student outcomes, the simple risk-factor accumulation strategy may fail to produce sub- 
groups with distinctly different academic trajectories. Therefore, in order to distinguish more 
effectively among subgroups of students who, in the absence of the program, would have ex- 
perienced distinctly different outcomes, the Career Academies Evaluation employed an impu- 
tation strategy for identifying subgroups. This is referred to throughout this report as a regres- 
sion-based subgroup strategy. 



III. Re gression-Based Subgroup Strateg y 

A. Overview of the Approach 

The basic idea behind the regression-based subgroup strategy is to build on the opportu- 
nity created by the random assignment experimental design in order to identify the relationships 
between background characteristics and student failure in the absence of the Academy interven- 
tion. Based on these relationships, one then identifies the characteristics of the students who, in 
the absence of the program, are most likely to drop out of high school. 

The regression-based strategy involves three steps. The first step is to use multiple regres- 
sion to estimate the relationship between several background characteristics measured at the time 
students applied to the Academy and the probability that they would drop out of high school be- 
fore the end of the 12 th grade. The background characteristics included in the Career Academies 
Evaluation are: 

• average daily attendance in the year the student applied for an Academy; 

• grade point average for the year the student applied for an Academy; 

• the number of credits earned toward graduation in the year the student applied 
for an Academy; 

• whether the student was overage for grade when entering the Academy; 

• whether the student had a sibling who dropped out of high school; and 

• whether the student had transferred schools two or more times beyond the 
typical school transitions. 

The goal of this analysis is to capitalize on the experimental design and estimate the rela- 
tionships between background characteristics and dropping out of high school in the absence of 
access to an Academy. The random assignment research design ensures that the non-Academy 
group provides the best counterfactual for what would have occurred to students in the absence 
of access to an Academy. Thus, the non-Academy group was used as the basis for this regression. 
Table B.l presents the results of this regression analysis. The first column of parameter estimates 
reflects the relationship between the dropout rate and a unit change in the background character- 
istics. Numbers in the second column are standardized to reflect the relationship between the 
dropout rate and a standard deviation change in the background characteristics. As the table sug- 
gests, all the characteristics included in this regression model are statistically significant and are 
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Table B.l 



Career Academies Evaluation 



Relationship Between Baseline Characteristics and the Probability of Dropping Out 
of High School Among Non-Academy Students 



Baseline Characteristic 


Coefficients 

Unstandardized Standardized 


Sibling dropped out 


0.08 *** 


0.03 *** 




(0.03) 


(0.01) 


Overage for grade 


0.06 ** 


0.02 ** 




(0.03) 


(0.01) 


Transferred schools 2 or more times 


0.07 *** 


0.03 *** 




(0.03) 


(0.01) 


Attendance rate in year of random assignment 


-0.01 *** 


-0.04 *** 




(0.00) 


(0.01) 


Credits earned in year of random assignment 


-0.05 *** 


-0.05 *** 




(0.01) 


(0.01) 


Grade point average in year of random assignment 


-0.03 * 


-0.02 * 




(0.02) 


(0.01) 


Intercept 


0.94 *** 


0.12 *** 




(0.14) 


-(0.14) 


R-squared 


0.10 


0.10 


Sample size 


763 


763 



SOURCES: MDRC calculations from Student Baseline Questionnaire Database and Student School Records 
Database. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for background 
characteristics of sample members. Rounding may cause slight discrepancies in calculating differences. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In both 
cases, statistical significance levels are indicated as *** = 1 percent; ** = 5 percent; *=10 percent. 



related to the probability that students would drop out of high school before the end of the 
12 th grade. 1 

The second step in this analysis is to combine the coefficients from the regression esti- 
mates for the non-Academy sample with the background characteristics of each individual in 



'Other specifications of this model were tried. However, through an informal process of model specification, 
this six-variable model was found to be the most sensible and effective. The estimates (below) of the potential dis- 
tortion caused by the regression-based approach do not take into account any effects of the model specification pro- 
cess on the impact estimates. 
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both the Academy and the non- Academy groups. In other words, the coefficient estimates from 
the regression are used as weights multiplied by the relevant measured background characteris- 
tics of each individual. The weighted sum of these characteristics yields an index indicating the 
probability of dropping out of high school. This is referred to as the risk index, and it provides a 
basis for ranking sample members according to the predicted probability that they would drop 
out of high school. 

For example, the parameter estimate associated with having a sibling who dropped out of 
school is .08 (that is, controlling for other background characteristics, students in the evaluation 
who had a sibling who already dropped out of high school were predicted to be 8 percentage 
points more likely to drop out of high school). Therefore, students with siblings who dropped out 
had .08 added to the index measuring their own risk of dropping out. By the same token, the re- 
gression estimates indicate that some characteristics are negatively correlated with dropping out. 
The weights assigned to these characteristics were multiplied by individual attributes and sub- 
tracted from the risk index. 

In the third step of this regression analysis, the Academy and non-Academy students are 
divided into three subgroups based on the risk index. Following is a brief definition of each of 
the three risk subgroups. 

• The high-risk subgroup: the students in the Academy and non-Academy 
groups with the combination of characteristics yielding scores at or above 
the 75 th percentile of scores on the risk index (that is, those with the high- 
est likelihood of dropping out) 

• The low-risk subgroup: the students in the Academy and non-Academy 
groups with the combination of characteristics yielding scores at or below 
the 25 th percentile of scores on the risk index (that is, those with the lowest 
likelihood of dropping out) 

• The medium-risk subgroup: the remaining students in the Academy and 
non-Academy groups (approximately 50 percent of the study sample) with 
a mix of characteristics yielding scores between the 25 th and 75 th percentile 
on the risk index (that is, indicating they were not particularly likely to 
drop out but not necessarily highly engaged in school) 2 

B. Strengths of the Regression-Based Strategy 

There are several important advantages to the regression-based strategy for defining sub- 
groups. First, it incorporates factors which are both conceptually and empirically related to stu- 
dents’ risk of dropping out of high school. At the same time, because these characteristics were 
measured prior to students’ random assignment to the Academy and non-Academy groups, they 
are exogenous to the Academy treatment. In other words, while the background characteristics 
used to create the subgroups were correlated with the likelihood of dropping out, these charac- 
teristics did not influence the selection of students into the Academy group. 



2 The 25 th and 75 th percentile cutoffs were based on the distribution of the risk index among the non-Academy 
students. 



An important question for such an impact analysis is whether, within each subgroup, the 
random assignment research design is preserved. In other words, are there systematic differences 
between the background characteristics of the Academy and non-Academy students within each 
subgroup? To test this, a set of background characteristics is regressed against a dummy variable 
indicating whether the student was assigned to the Academy group. Appendix A presents the re- 
sults of this analysis, which revealed that while there are a few differences between the back- 
ground characteristics of Academy and non-Academy students within each subgroup, f-tests 
failed to reject the hypothesis that there are no overall systematic differences between the back- 
ground characteristics of the Academy and non-Academy students. This suggests that the random 
assignment research design was preserved within each subgroup. In other words, the existing dif- 
ferences are not greater than those which would be expected to occur by chance. 

A second strength of this approach is that it incorporates the fact that the relationships 
between “risk factors” and student outcomes vary, depending on the background characteristic. 
For example, the coefficient estimates suggest that the effect of the number of credits earned in 
the year prior to random assignment and the effect of baseline attendance on the dropout rate are 
each at least twice as large as the effect of a student’s baseline grade point average or whether a 
student was overage for grade. 3 Basing the subgroup definitions on these relationships allows 
these differences to be factored into the classification of students into the three risk subgroups. 
For example, these regression estimates suggest that an average student who had a sibling who 
had dropped out and who was overage for grade would have approximately a 24 percent chance 
of dropping out of high school before the end of the 12 th grade. However, if that same student 
also had 98 percent attendance and was about a standard deviation above the average in terms of 
credits earned, he or she would have only a 16 percent chance of dropping out. 4 

Moreover, the regression-based strategy is capable of incorporating variation across stu- 
dents along continuous variables such as attendance and grade point average. Less flexible 
strategies that fail to incorporate these factors would not be as effective at distinguishing among 
students at different levels of academic risk. For example, an otherwise average student with per- 
fect attendance (that is, 100 percent) has a 9 percent chance of dropping out; a similar student 
with an attendance rate of 95 percent has a 12 percent chance of dropping out; a student with a 90 
percent attendance rate has a 15 percent chance of dropping out; and one with 85 percent atten- 
dance has a 17 percent chance of dropping out. In other words, there appears to be meaningful 
variation in the probability of dropping out that would not be captured by a simple categorical 
measure of attendance. The regression-based subgroup strategy captures such variation and in- 
corporates it into the assessment of each student’s risk of school failure. 

The third and perhaps most important strength of the regression-based strategy is that it 
effectively identifies students with distinct academic trajectories. Figure B.l presents the dropout 
rates for Academy and non-Academy students, as well as the difference between their dropout 



3 Note that these coefficients have been standardized to reflect the effect of a standard deviation change in the 
independent variable on the dropout rate, thus making the coefficient estimates directly comparable with one an- 
other. 

4 The predicted probability of dropping out for the average student was estimated by multiplying the mean val- 
ues of the independent variables among the students in the study sample by the coefficients in Table B.l. The esti- 
mated probabilities for students with the hypothesized characteristics were estimated by substituting the hypothe- 
sized values for the mean values where appropriate. 
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Figure B.l 

Impact of Career Academies on Dropout Rate, by Deciles of the 
Regression-Based Risk Index 
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rates, at 10 percentile intervals on the regression-based risk index. The black bars represent the 
percentage of non- Academy students who dropped out of high school, and the white bars repre- 
sent the percentage of Academy students who did so. The striped bars represent the difference 
between these two groups, that is, the impact of the Academy treatment on dropout rates. The 
pattern in this figure suggests that the risk index very effectively differentiates among students 
with different academic trajectories, and that the relationship between risk and the impact of 
Academies on dropout rates is not isolated to a small segment of the student population. 

The figure indicates that both the risk of dropping out and the impact of the program on 
this outcome generally increase with the percentiles of the risk index. In particular, the dropout 
rate among the non-Academy group appears to increase steadily with the percentiles of the risk 
index, and it grows sharply after the 70 th percentile. The impact on the dropout rate follows 
essentially the same pattern. From the 30 th percentile through the 90 th , the difference between 
the Academy and non-Academy groups becomes increasingly negative. The magnitude of this 
reduction in dropout rates appears to increase dramatically after the 70 th percentile, and then it 
shrinks slightly among students above the 90 th percentile of risk. This pattern suggests that, for 
the individuals with low to moderate risk of dropping out, the impact of the program on drop- 
out rates appears to be rather negligible. However, as the risk of academic failure becomes 
more serious, the impact of the Academy approach appears to grow. Finally, for those at great- 
est risk, the impact on dropout rates is substantial, but it is not as great as for those who are 
slightly less at risk. 

In short, this graph illustrates that the regression-based strategy is quite effective at dif- 
ferentiating among students with different degrees of Academic risk, and that the impact of the 
Academies on the dropout rate is strongly related to this definition of academic risk. 

Table B.2 illustrates that the regression-based strategy is effective at differentiating 
among students with different trajectories across a variety of school outcomes, and that it is more 
effective than the risk-factor accumulation strategy for making these distinctions. The table pres- 
ents several key measures of student performance during high school for the non-Academy stu- 
dents within each risk subgroup. The first panel of the table presents non-Academy outcome lev- 
els and estimated impacts based on the risk-factor accumulation approach, and the second panel 
presents these estimates based on the regression-based approach to defining subgroups. As the 
table illustrates, the regression-based strategy does a better job of making distinctions among 
students with different levels of academic risk. 

According to the estimates generated by the regression-based approach, while 32 percent 
of the non-Academy students in the high-risk subgroup dropped out of high school before the 
end of the 12 th grade, 8 percent in the medium-risk subgroup dropped out, and less than 3 percent 
in the low-risk subgroup did so. Moreover, while only 27 percent of the non-Academy students 
in the high-risk subgroup earned enough credits to graduate from high school, 65 percent in the 
medium-risk subgroup and 75 percent in the low-risk subgroup did so. Similar patterns were 
found for most other measures as well. This indicates that, without access to a Career Academy, 
the students in the different risk subgroups would have had substantially different outcomes. 

Table B.2 also provides outcome levels and estimated impacts for subgroups based on the 
risk-factor accumulation approach. Not surprisingly, these estimates are not as distinct from one 
another as the estimates generated by the regression-based approach. For example, 22 percent of 



Table B.2 

Career Academies Evaluation 

Selected Outcomes Among Non-Academy Students, 
by Risk Subgroups Defined Using Risk-Factor 
Accumulation and Regression-Based Index 





Accumulation Approach 


Regression-Based Approach 


Outcome 


Non-Academy 

Outcomes (%) Impact 


Non-Academy 

Outcomes (%) Impact 



High-risk subgroup 



Dropped out of high school 


27.4 


-5.6 * 


32.3 


-11.4 *** 


Earned credits to graduate 


34.1 


10.0 ** 


27.0 


12.8 *** 


Completed basic academic core 


9.9 


9.1 * 


5.6 


8.0 * 


Reported any negative risk-taking 


36.9 


-5.9 


38.9 


-3.8 


Reported positive youth development 


56.8 


8.7 * 


54.9 


8.0 


Medium-risk subgroup 










Dropped out of high school 


9.3 


-2.0 


7.9 


0.9 


Earned credits to graduate 


66.2 


-0.6 


64.8 


0.8 


Completed basic academic core 


31.0 


-2.7 


30.3 


-1.2 


Reported any negative risk-taking 


23.1 


0.5 


25.7 


-2.2 


Reported positive youth development 


67.5 


4.1 


69.7 


1.6 


Low-risk subgroup 










Dropped out of high school 


4.2 


0.5 


2.8 


-1.2 


Earned credits to graduate 


69.5 


8.4 ** 


74.8 


12.9 ** 


Completed basic academic core 


33.9 


0.9 


36.6 


4.5 


Reported any negative risk-taking 


22.0 


-2.1 


15.8 


-1.0 


Reported positive youth development 


75.4 


0.8 


75.5 


6.3 



SOURCES: MDRC calculations from Career Academies Evaluation Student School Records Database and 12th 
Grade Survey Database. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for background characteristics 
of sample members. Rounding, may cause slight discrepancies in calculating differences. 

A two-tailed t-test was applied to differences between the Academy and non-Academy groups. In both cases, 
statistical significance levels are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent. 



students in the “high-risk” subgroup dropped out of high school before the end of the 12 th grade, 
compared with 7 percent in the “medium-risk” subgroup and 5 percent in the “low-risk” sub- 
group. Moreover, 44 percent in the “high-risk” subgroup earned enough credits to graduate, 
compared with 66 and 78 percent in the “medium-risk” and “low-risk” subgroups, respectively. 

These patterns in outcome levels among students who weren’t exposed to the Academy 
treatment suggest that the regression-based strategy is the more effective means for defining sub- 
groups of Career Academy students with substantially different academic trajectories. Interest- 
ingly, the impact estimates suggest that while the estimates generated by the regression-based 
approach tend to be somewhat larger, their pattern is similar to the pattern of estimates based on 
the risk-factor accumulation model. For example, among students in the high-risk subgroup, both 
the regression-based approach and the risk-factor accumulation approach found that Academies 
significantly reduced dropout rates, increased credits earned toward graduation, and increased the 
percentage of students who completed a core academic curriculum. So while the regression- 
based approach was more effective at identifying students who, in the absence of the Academy 
treatment, would have had substantially different outcomes, it did not distort the basic pattern of 
impacts generated by the experiment. 

C. Potential Limitations of the Regression-Based Approach 

While the regression-based strategy is more effective than the risk-factor accumulation 
strategy at identifying students who were likely to experience different academic trajectories in 
the absence of the Academy, it has some potentially important limitations. First, although it is 
more systematic, it is also less straightforward than the risk-accumulation strategy in terms of 
the manner by which subgroups of students might be identified by school administrators. In 
particular, to the extent that these subgroup findings might be used to target program resources 
toward particular individuals, the subgroups defined using the regression-based strategy might 
be more difficult to identify than subgroups based on a simple accumulation approach. While 
it is unclear that the implications of the findings from this particular study suggest that target- 
ing would be advantageous, such thinking may be a factor when applying this strategy to the 
study of programs in which the implications of targeting are less ambiguous. Although it is not 
discussed in this appendix, the regression-based approach can be applied in a practical way and 
may, in fact, be a more systematic way of targeting resources toward students most likely to 
benefit from them. For example, this type of approach has been used in research designed to 
develop approaches for the targeting of benefits and associated employment services to work- 
ers eligible for unemployment insurance as well as for targeting employment resources to in- 
dividuals in welfare-to-work programs. In particular, several of these programs have used his- 
torical data to estimate the relationship between background characteristics and policy-relevant 
outcomes, and then to combine these estimates with individual characteristics in order to pre- 
dict outcomes and target services. This has been done in welfare-to-work programs in Michi- 
gan as well as in unemployment programs in Michigan, New Jersey, and Washington 
(O’Leary, Decker, and Wadner, 1998; Eberts, 1997). 

A more important potential limitation of the regression-based subgroup strategy is related 
to the manner in which the strategy generates weights relating background characteristics to risk. 
In short, theoretically, the strategy has the potential to overstate any positive impacts of the pro- 
gram on the high-risk subgroup and to overstate the magnitude of any negative impacts on the 
low- and medium-risk subgroups. 



The problem has its genesis in the fact that the regression parameter estimates that are 
used as weights to translate student characteristics into academic risk are the result of estimates 
that are specific to the non-Academy group. In a sample from any population, estimated regres- 
sion coefficients reflect both the relationships that exist in the population and a random element 
that is specific to that sample. In other words, on average, each coefficient from such a random 
sample is unbiased. However, it is highly unlikely that, in any given sample, the estimated re- 
gression coefficient will exactly equal the true regression coefficient from the entire population 
from which that sample is drawn. Therefore, the regression estimates from the non-Academy 
group include some random error that is particular to the non-Academy group and that is corre- 
lated with the outcome in question — in this case, whether or not a student dropped out before 
the end of the 12 th grade. 

For example, Equation 1 is a simple regression predicting dropout from a set of back- 
ground characteristics for a sample of students drawn from the population of students who ap- 
plied to a Career Academy: 



Y i =d + $X t +e t 

where: 



( 1 ) 



Y, = 1 if student i dropped out; 0 otherwise; 

X { = 1 if student i had ever been held back; 0 otherwise (this could be any impor- 

tant background characteristic); 

ex = the intercept term, that is, the average outcome ( I" ) among those where X = 0; 
and 

P = the estimated relationship between X t and Y t , that is, the estimated effect of 
X i on the probability that a student drops out of high school. 



In this case, it would also be true that: 



P = P+P, 



( 2 ) 



where: 



^ - the true relationship between X and Y in the population from which our sample 
was drawn; and 

P* = the difference (or error) between the relationship between X and Y in the 
population from which the sample was drawn and the relationship between X 
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and Y in the sample, that is, the element of the estimate which is idiosyncratic 
to the particular sample. 

While p is a characteristic of the population and does not change from sample to sample, 
P s is particular to the sample upon which the regression is estimated, and it will vary from sam- 
ple to sample. As a result, while p never changes, (3 will vary from sample to sample. Further- 
more, it is also highly unlikely that the random error in a coefficient estimated from one sample 
drawn from a population will be exactly the same as the random element in any other sample 
drawn from the same population. 

The students in the Career Academies evaluation sample were assigned to the Academy 
or non-Academy groups at random; therefore, one can have a high degree of confidence that 
there are no systematic differences between these two groups in terms of observable or unobserv- 
able characteristics. They can be thought of as two random samples drawn from the same popu- 
lation of students at these sites who applied to and were eligible for the Career Academies. While 
the program may have changed the relationships between background characteristics and the 
probability of dropping out, the underlying relationship between background characteristics and 
the likelihood of dropping out in the absence of the Academy intervention ( P ) is the same for 

these two groups. 

However, even in the absence of the Academy program, it is unlikely that the estimated 
coefficients relating the background characteristics to the dropout rate among the students who 
ended up in the program group would have been exactly the same as those in the non-Academy 
group. In other words, while the underlying relationship between background characteristics and 
the probability of dropping out ( p ) would not vary across these two samples, the idiosyncratic 
element (or error term) of the estimated relationship ( PJ, and therefore the estimated relation- 
ship itself ( (3 ), would vary. 

Therefore, it is highly unlikely that the estimated relationship between background char- 
acteristics and dropout would have been exactly the same among the Academy group as it was 
among their non-Academy group counterparts. Because the regression weights were generated 
from the non-Academy group, the regression-based strategy might more accurately distinguish 
among students with different levels of academic risk for this group than it does for the Academy 
group. In other words, the risk index might distinguish different levels of risk more effectively 
among non-Academy students that it does among Academy group students. 

This creates the possibility that, although their observable characteristics were the same, 
students in the “high-risk” non-Academy group were actually more at risk than students in the 
“high-risk” Academy group. It also creates the possibility that students in the “low-risk” non- 
Academy group were actually less at risk than students in the “low-risk” Aademy group. 

To the extent that this occurred, it would result in overstating positive impacts for the 
high-risk subgroups and overstating the magnitude of negative impacts for the “low-risk” sub- 
groups. However, as the next section will reveal, the magnitude of this potential distortion can be 
estimated. Furthermore, the magnitude of the distortion appears to be minimal, and it is not large 
enough to have a meaningful effect on the overall pattern of impact estimates. 
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IV. Magnitude of Potential Distortion in the Regressions-Based Approach 

In order to understand whether this potential limitation outweighs the analytic advantages 
of the regression-based approach discussed earlier, it is important to estimate the magnitude of 
the potential distortion. 

Theoretically, in order to estimate the magnitude of this distortion, one would like to 
compare the outcomes of the students within each risk category in the non-Academy group with 
what would have been observed among the Academy students in the same risk subgroup in the 
absence of the treatment. However, because the Academy group received the treatment and the 
treatment may have actually affected these outcomes, this comparison cannot be made. The ideal 
basis for such a comparison would be a second non-Academy group that was neither used in or- 
der to estimate the dropout regression nor exposed to the program. In the absence of any distor- 
tion, one would expect that, within each risk subgroup, the outcomes for the students in this sam- 
ple would be identical to the outcomes for these in the original non-Academy group. Therefore, 
any differences between outcomes for these students and outcomes for the original non-Academy 
group could be confidently attributed to the distortion created by the regression-based strategy. 

Although a second non-Academy group for this study is not available, a strategy for esti- 
mating the potential distortion in the original estimates is to use bootstrap sampling in order to 
simulate a second sample. Bootstrap sampling is commonly used to generate estimates of stan- 
dard errors and other population characteristics from relatively small samples (Stine, 1990). It 
rests on the assumption that the sample from which the observations are drawn is representative 
of the population as a whole. In this case, to the extent that the initial non-Academy group can be 
thought of as representative of the population of students from whom the evaluation sample was 
drawn, bootstrap sampling procedures can be used to simulate new samples of non-Academy 
group students. Within each subgroup, these samples can be used in order to compare the out- 
comes for the students on whom the dropout regression was based with the outcomes for a sam- 
ple of students who were not included in this regression. These differences would constitute a 
reliable estimate of the distortion created by the regression-based subgroup strategy. 

The mechanics of this process are as follows: 

1 . Use a random number generator to draw a bootstrap sample of students the size of the 
original non-Academy group, sampling with replacement the observations from the 
original non-Academy group sample. 

a. Use a random number generator to select an observation from the original 
non-Academy group. 

b. Copy that observation to a new data set. 

c. Replace that observation into the sampling frame from which it was drawn 
(the original non-Academy group sample). 

d. Repeat steps a through c until the new sample equals the size of the original 
non-Academy group (n=805 times). This sample will be referred to as the 
model group. 
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This creates a sample which is the same size as the original non-Academy 
sample and which, theoretically, is drawn from the same population. 5 How- 
ever, this sample is not the same as the non-Academy group, because steps a 
through d typically create a sample which omits several ob: jrvations from the 
original sample and creates multiple copies of other observs ions. 

2. Use this bootstrap sample to estimate the relationship between the six back- 
ground characteristics used to define academic risk and the probability that a 
student will drop out of high school prior to the end of the 12 th grade. 

3. Repeat steps la through Id to draw (with replacement) a second bootstrap 
sample, the size of the Academy group, from the original non-Academy sam- 
ple. This sample will be referred to as the non-model group. 

4. Repeat steps la through Id once more, this time drawing from the Academy 
sample, to produce a bootstrap sample of students from each risk subgroup 
who received the Academy treatment. This sample will be referred to as the 
simulated Academy group. 

5. Apply the coefficients from the regression model to the background charac- 
teristics of the individuals in all three bootstrap samples in order to create the 
risk index. 

6. Use the 25 th and 75 th percentiles of the risk index in the first bootstrap sample 
(the model group) in order to divide the samples into high-, medium-, and 
low-risk subgroups. 

7. Compare the average outcomes from the model group with those from the 
second bootstrap sample (the non-model group). The difference between the 
two groups represents the distortion created by the regression-based strategy. 

8. Repeat steps 1 through 7 another 200 times. The average difference across 
these iterations between the subgroup outcomes for the model group and the 
non-model group provides a bootstrap estimate of the potential distortion cre- 
ated by the regression-based subgroup strategy. The average levels across 
these iterations among the simulated Academy group represents a bootstrap 
estimate of the outcome levels among the Academy students. 

Table B.3 presents the results of this estimation process for five key outcomes. The num- 
bers in this table represent the average outcomes of 200 iterations of the bootstrap process de- 
scribe above. As such, they are intended to simulate what one would expect to observe if one re- 
peated the experimental analysis contained in the report 200 times, with 200 different samples 
from the same population. The first column of the table presents the average outcomes among 
students from the bootstrap samples upon which the dropout regression was estimated (the model 
group). The numbers in this column represent the outcome levels one would expect to observe as 



5 In particular, this replaces the unknown theoretical distribution of the population from which the non-Academy 
group is drawn with the empirical distribution of the non-Academy sample itself. 
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Table B.3 

Career Academies Evaluation 



Outcome Levels for Bootstrap Control Samples and Program Group, 

by Risk Subgroups 



Outcome 


Model Non-Model 
Group Group 

(%) (%) 


Model 

Minus 

Non-Model 


Program 

Group 

(%) 


Program 

Minus 

Model 


Program 

Minus 

Non-Model 


High-risk subgrouo 














Dropped out of high school 


31 


30.3 


0.7 ** 


20.7 


-10.3 


-9.6 


Earned credits to graduate 


28.5 


28.6 


-0.1 


40.2 


11.7 


11.6 


Completed basic academic core 


7.3 


7.2 


0.1 


15.3 


8 


8.1 


Reported any negative risk-taking 


39.5 


39 


0.5 


32.9 


-6.6 


-6.1 


Reported positive youth development 


55.4 


56.1 


-0.7 * 


64.5 


9.1 


8.4 


Medium-risk subgroup 














Dropped out of high school 


8.4 


8.9 


-0.5 *** 


9 


0.6 


0.1 


Earned credits to graduate 


63.4 


63.4 


0 


65.6 


2.2 


2.2 


Completed basic academic core 


28.9 


29 


-0.1 


28.2 


-0.7 


-0.8 


Reported any negative risk-taking 


25.3 


25.1 


0.2 


24.2 


-1.1 


-0.9 


Reported positive youth development 


68.6 


68.8 


-0.2 


70.6 


2 


1.8 


Low-risk subgroup 














Dropped out of high school 


2.8 


2.9 


-0.1 


2.2 


-0.6 


-0.7 


Earned credits to graduate 


75.8 


76.1 


-0.3 


84.9 


9.1 


8.8 


Completed basic academic core 


36.7 


36.6 


0.1 


39 


2.3 


2.4 


Reported any negative risk-taking 


16.8 


16.7 


0.1 


15.7 


-1.1 


-1 


Reported positive youth development 


77.2 


76.8 


0.4 


80.3 


3.1 


3.5 



SOURCES: MDRC calculations from Career Academies Evaluation Student School Records Database and 12th Grade 
Survey Database. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for background characteristics of 
sample members. Rounding may cause slight discrepancies in calculating differences. 

A two-tailed t-test was applied to differences between the model and non-model groups. In both cases, statistical 
significance levels are indicated as *** = 1 percent; ** = 5 percent; * = 10 percent. 

a result of the regression-based approach among the sample of non- Academy students on whom 
the regression was fit. 

Column 2 of Table B.3 presents the average outcomes among students from the bootstrap 
samples which were not used for this regression (the non-model group). The numbers in this col- 
umn represent the pattern of outcomes one would expect to observe if one had a sample of non- 
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Academy students who were not the basis for the regression model but for whom the coefficients 
from the regression-based strategy were combined with individual characteristics in order to es- 
timate the risk of school failure. 

The third column of Table B.3 presents the differences between the two averages for the 
model and non-model groups. Because the second column of estimates is not affected by the po- 
tential distortion described above, these numbers represent the estimate of the potential distortion 
created by the regression-based strategy for each outcome. 

The fourth column of Table B.3 presents the average outcomes for the high-, medium-, 
and low-risk subgroups from the simulated Academy (program) group. The fifth column presents 
the average differences between the simulated Academy group and the model group from column 

1 . This represents a bootstrap estimate of the program impact. The sixth column presents the av- 
erage differences between the simulated Academy group and the non-model group from column 

2. This represents a bootstrap estimate of the program impact, absent any distortion created by 
the regression-based subgroup strategy. 

The estimates in Table B.3 suggest that the magnitude of the distortion created by the re- 
gression-based subgroup strategy is not large enough to have a meaningful effect on the pattern 
of impacts described in the report. In particular, for each of the outcomes in this table, the esti- 
mated distortion appears to be less than 1 percentage point. For example, the first row of the ta- 
ble presents the bootstrap estimates of the dropout rate for the high-risk subgroup. Inasmuch as 
whether or not a student dropped out of high school was the dependent variable in the regression 
used to define the subgroups, the potential magnitude of the distortion should be largest with re- 
spect to that outcome. However, the estimate in this row suggests that the potential distortion in 
the impact estimate is seven-tenths of 1 percentage point. In particular, across 200 replications, 
the average dropout rate for the high-risk sample from the model group is 31 percent, while the 
average for the sample that was not used to estimate the regression (from the non-model group) 
is 30.3 percent, a difference of .7 percentage points. 6 

Columns 4 and 5 of Table B.3 indicate that subtracting the potential distortion does not 
result in a meaningfully different estimate of the program impact. In particular, the estimate of 
the impact and the estimate of the impact minus any potential distortion appear to be within 
rounding error of one another. Moreover, the other estimates in this table reveal a similar pattern. 
The estimated distortion is never larger than seven-tenths of a percentage point, and the pattern 
of effects in the impact estimates is not substantially different from the pattern of effects in the 
column estimates that account for the distortion. This suggests that, while the regression-based 
subgroup strategy has theoretical limitations, the limitations do not have any meaningful effect 
on the pattern of impacts presented in this report. 



6 An alternate estimate of the distortion was generated by performing what might be called a randomization test. 
This entailed taking the entire evaluation sample, including Academy and non-Academy students, and randomly 
assigning them to two groups. The dropout regression was then estimated within one group, and the coefficients 
were used to generate an index and divide the sample into risk categories in both groups. The difference between the 
outcomes for these groups would represent an alternative estimate of the distortion. After performing this process 
200 times, it was found that this alternative method yielded a pattern of estimated distortion similar to that produced 
by the initial method. In particular, the estimated distortion on the dropout variable was 1.3 percentage points, and 
the estimated distortion on all other variables was smaller than that. 
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The asterisks in the table indicate the results of statistical significance tests regarding the 
differences between the model and non-model groups. They suggest that, across the five out- 
comes and three subgroups considered, the estimated distortion created by the regression-based 
subgroup strategy was statistically significant in only three cases. In particular, for the high-risk 
subgroup, the estimated distortion created with respect to the dropout variable and the percentage 
of students who participated in positive youth development activities was significantly different 
from zero. For the medium-risk subgroup, the distortion created regarding the dropout rate was 
also statistically significant. The estimated distortion across all other outcomes was not signifi- 
cantly different from zero. 

This pattern, combined with the magnitude of the effects, suggests two conclusions. First, 
the estimated distortion created by the regression-based subgroup strategy appears to converge 
around some non-zero number, but that effect does not appear to be large enough to affect the 
basic pattern of impacts. Second, the distortion appears to be restricted mainly to the outcome 
variable that was the basis for defining the subgroups, and it was concentrated within the high- 
risk subgroup. 



V. Conclusions 

The evidence and discussion in this appendix strongly support the idea that accounting 
for the heterogeneity of students in the Career Academies Evaluation is an important element of 
any strategy designed to assess the impact of the Academies on the diverse group of students 
they serve. Impact estimates which aggregate results across students with different academic tra- 
jectories conceal a substantial amount of variation across students in the effects of the Academies 
on key outcomes. Therefore, in order to assess the effects of Career Academies more sensitively, 
it is necessary to develop a strategy for differentiating among students who, in the absence of the 
Academy treatment, would experience different academic outcomes. 

Traditional approaches toward defining subgroups go part of the way toward differenti- 
ating among students with different academic trajectories. However, the experimental design pre- 
sent in the Career Academies Evaluation provides a rare opportunity to improve on these strate- 
gies by estimating the relationship between student characteristics and the likelihood of school 
failure in the absence of the Academy treatment. 

This regression-based approach offers a number of distinct advantages over its alterna- 
tives, and its potential limitations are highly unlikely to change the pattern of any of the find- 
ings. The regression-based approach takes multiple factors into account, weighting them ac- 
cording to the strength of their effect on student failure. It also allows the use of all relevant 
variation in student characteristics in order to estimate risk, as opposed to classifying students 
on the basis of arbitrary cutoffs in otherwise continuous measures of risk. Most important, it is 
a highly effective strategy for identifying students who, in the absence of the Academy inter- 
vention, would have had substantially different outcomes. As a result, it reveals differences in 
the effects of Career Academies that would be masked by impacts which are averaged across 
the entire population of Academy students — and would be at least partly masked by tradi- 
tional approaches to defining subgroups. 
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The major drawback of the regression-based strategy is that it has the potential to gen- 
erate a distortion in the impact estimates that would overstate the impact of the Academies on 
students in the high-risk subgroup. However, the best estimates of the potential distortion in 
impact estimates suggest that its magnitude is negligible. In particular, the estimates suggest 
that the distortion, at it largest, is seven-tenths of a percentage point. Moreover, any distortion 
which exists appears to be concentrated within the high-risk subgroup and to be restricted pri- 
marily to one outcome. In other words, both the magnitude and pattern of distortion suggest 
that this phenomenon is neither large nor pervasive enough to affect the overall pattern of im- 
pacts presented in the report. 
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Recent Publications on MDRC Projects 



Note: For works not published by MDRC, the publisher’s name is shown in parentheses. A complete publications 
list is available from MDRC and on its Web site (www.mdrc.org), which also contains copies of MDRC’s 
publications. 

Education Reform 

Career Academies 

The largest and most comprehensive evaluation of a school-to-work initiative, this 1 0-site study examines a 

promising approach to high school restructuring and the school-to-work transition. 

Career Academies: Early Implementation Lessons from a 1 0-Site Evaluation. 1996. James Kemple, JoAnn Leah 
Rock. 

Career Academies: Communities of Support for Students and Teachers — Emerging Findings from a 1 0-Site 
Evaluation. 1997. James Kemple. 

Career Academies: Building Career Awareness and Work-Based Learning Activities Through Employer 
Partnerships. 1999. James Kemple, Susan Poglinco, Jason Snipes. 

School-to-Work Project 

A study of innovative programs that help students make the transition from school to work or careers. 

Home-Grown Lessons: Innovative Programs Linking School and Work (Jossey-Bass Publishers). 1995. Edward 
Pauly, Hilary Kopp, Joshua Haimson. 

Home-Grown Progress: The Evolution of Innovative School-to-Work Programs . 1997. Rachel Pedraza, Edward 
Pauly, Hilary Kopp. 

Project Transition 

A demonstration program that tested a combination of school-based strategies to facilitate students’ transition from 

middle school to high school. 

Project Transition: Testing an Intervention to Help High School Freshmen Succeed. 1999. Janet Quint, Cynthia 
Miller, Jennifer Pastor, Rachel Cytron. 

Equity 2000 

Equity 2000 is a nationwide initiative sponsored by the College Board to improve low-income students’ access to 

college. The MDRC paper examines the implementation of Equity 2000 in Milwaukee Public Schools. 

Getting to the Right Algebra: The Equity 2000 Initiative in Milwaukee Public Schools. 1999. Sandra Ham, Erica 
Walker. 



Other Programs for Youth 

The JOBSTART Demonstration 

A test of a program combining education, training, support services, and job placement for very disadvantaged 
young high school dropouts. 

JOBSTART: Final Report on a Program for School Dropouts. 1993. George Cave, Hans Bos, Fred Doolittle, Cyril 
Toussaint. 

The Career Beginnings Evaluation 

An evaluation of a program that seeks to increase college attendance and improve job quality among disadvantaged 
high school students. 

Career Beginnings Impact Evaluation: Findings from a Program for Disadvantaged High School Students. 1990. 
George Cave, Janet Quint. 



Note: For works not published by MDRC, the publisher’s name is shown in parentheses. 
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The Youth Incentive Entitlement Pilot Projects (YIEPP) Demonstration 

A test of a school-conditioned job guarantee for low-income youth. 

Lessons from a Job Guarantee : The Youth Incentive Entitlement Pilot Projects. Monograph. 1984. Judith Gueron. 

Teen Parents on Welfare 

Teenage Parent Programs: A Synthesis of the Long-Term Effects of the New Chance Demonstration, Ohio ’s 
Learning, Earning, and Parenting (LEAP) Program, and the Teenage Parent Demonstration (TPD). 1998. Robert 
Granger, Rachel Cytron. 

Ohio's LEAP Program 

An evaluation of Ohio’s Learning, Earning, and Parenting (LEAP) Program, which uses financial incentives to 
encourage teenage parents on welfare to stay in or return to school. 

LEAP: Final Report on Ohio s Welfare Initiative to Improve School Attendance Among Teenage Parents. 1997. 
Johannes Bos, Veronica Fellerath. 

New Chance Demonstration 

A test of a comprehensive program of services that seeks to improve the economic status and general well-being of 
a group of highly disadvantaged young women and their children. 

New Chance: Final Report on a Comprehensive Program for Young Mothers in Poverty and Their Children. 1997. 
Janet Quint, Johannes Bos, Denise Polit. 

Parenting Behavior in a Sample of Young Mothers in Poverty: Results of the New Chance Observational Study. 

1998. Martha Zaslow, Carolyn Eldred, editors. 
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Over the past quarter century, MDRC has worked in almost every state, all of the 
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